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ABSTRACT 

Two research questions are addressed in this report; 
What outcomes are appropriate and feasib3.e to use as criteria for 
evaluatinq vocational programs? and what procedures can be used for 
evaluating vocational education on the basis of specified outcome 
criteria? I'cllcwing brief reference tc earlier studies on vocational 
education outcomes, fifteen possible crtconjes are listed and 
discussed in terms of their importance, appropriateness, and 
feasibility for use in evaluation; improving basic educational 
skills, development of useful occupational skills, reducing the risk 
of unemplcviu€nt, acquiring world-cf-wcik knowledge, effect on 
educational commitment, development of leadership qualities, 
postsecondary educational v'^ogress, level of postschooi earning, 
satisfaction with current schcol experience, job-search time, 
satisf actoriness to emplovers, attractiveness of the commanity for 
industrial development, ercplcyment cppcttunities for minority 
workers, job placement in training-related fields, and development of 
self-help skill??. Fourteen essential elements of information for an 
evaluation study are then ider+ified and notes are provided for 
OF.-rationali2ing ^ach of the fifteen outcomes. Finally, a brief 
report is ftiven of a pilot study testirg one outcome (reducing the 
risk of UTjemplcyment for minority youth) carried out with vocational 
educat ion-^perscnnel in two states. Findings are summarized and 
recommendations made for' further research, development, and 
evaluaticp. activities. (JT) 
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FOREWORD 



Program evaluation Is a bailc component of good management as well as a legislative 
requlremont. Evaluation research can generate data to serve as a basis for Informed decisions on 
matters of policy, program design, and program operation. But what criteria should be used for 
evaluating vocational education? And what procedures offer promise for maximum returns from 
investments In evaluative research? 

These two questions are specifically addressed In this report, which carries the National 
Center's ongoing study of vocational education outcomes beyond the exploratory stage. (See 
Chapter I for titles of four earlier publications resulting from the outcomes study.) Presented 
here is a detailed discussion of (a) fifteen key outcomes of vocational education, (b) issues 
related to the "validation" of these outcomes, (c) operational procedures for conducting 
outcomes studies, and (d) some results from pilot testing these procedures with one of the 
hypothesized outcomes. The report concludes with a summary of lessons learned and 
recommendations offered for applying the findings and products of the study. 

Readers who may find this report of particular interest include vocational education 
evaluators, state and local vocational directors, sponsors of evaluation studies, policy analysts, 
members of the academic community, and people who advise and make decisions on vocational 
education policy. 

The National Center Is pleased to acknowledge the advice and assistance of several 
individuals who made important contributions to this study.' Serving as project consultants were 
Dr. Paul A. Games, Pennsylvania State University; Dr. John T. Grasso, West Virginia University; 
Dr. Douglas Sjogren, Colorado State University; Dr. Jesse 8. Clemmons, North Carolina 
Department of Education; and Jeanette F. McConaughy, Worthlngton, Ohio. Helpful suggestions 
were provided by Dr. Betsy Bo-ak Houser, Education Director of Fink and Kosecoff, Inc., of 
Santa t^/1onica, California, and Dr. Frederick Haddad, Connecticut Department of Education, both 
of whom reviewed a draft of the report. Dr. Ruth Hughes, Iowa State University, and Dr. George 
Copa, University of Minnesota, both members of the National Center's Evaluation Technical 
Advisory Panel, provided special assistance In the study. Kathleen A. Bolland contributed id3as 
and assisted in field work during the early stages of the study. The study was sponsored by the 
Bureau of Occupational and Adult Education, U.S. Office of Education. 

The National Center is grateful to Dr. Robert L. Darcy, Senior Research Specialist, for 
directing the outcomes study and preparing this report; and to Pamela J. Davis for her 
excoilence in typing the manuscript. 



Robert E. Taylor 
Executive Director 
The National Center for Research 
in Vocational rducation 
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I. BACKGROUND AND PERSPECTIVE 



Vocational educalion, along with other human service programs, can be evaluated on the 
basis of many different criteria end, indeed, diverse types of criteria. In an earlier report on the 
National Center's examination of vocational education outcomes (Darcy, 1979a) six different 
types of evaluation criteria were identified. These include (1) the context in which vocational 
education operates, (2) characteristics of the students enrolled in vocational programs, (3) the 
quantity and quality of resources used In vocational education, (4) program goals and objectives. 
(5) processes utilized for Instruction and related activities, and (6) the consequences or 
outcomes of the overall vocational education enterprise. 

The evaluation arena is broad and complex, and in vocational education there Is much to be 
seen and studied. Not everything can be done all at once. The particular terrain on which the 
present study focuses is outcomes. The inquiry proceeds on the assumption that outcomes are 
important and may even be the most important basis for evaluating vocational education; but 
outcomes are definitely not the only basis for evaluating vocational programs. 



A. ReMarch Questions Addressed 

Two research questions are specifically addressed in this report: 

• What outcomes are appropriate and feasible to use as criteria for evaluating vocational 
programs? 

• What procedures can be used for evaluating vocational education on the basis of 
specified outcome criteria? 

B. Relationship to Other Studies 

This report contains ideas and information based on what might be tei-ned Phase Two of a 
projected long-term examination of vocational education outcomes and th^jir correlates (Darcy, 
Bolland, and Farley, January 1979). 

Phase One was exploratory, focusing on conceptual Issues, reviewing the literature of 
outcomes evaluation, and seeking to expand awareness of the range and diversity of vocational 
education outcomes. This phase resulted in publication of the following documents; 

• Boiland, Kathleen A. Vocational Education Outcomes: An Evaluative Bibliography of 
Empirical Studies. Bibliography Series No. 49. Columbus: National Center for Research in 
Vocational Education, The Ohio State University. 1979. 

• Darcy. Robert L. Vocational Education Outcomes: Perspective for Evaluation. Research 
and Development Series No. 163, Columbus: National Center for Research in Vocational 
Education. The Ohio State University. 1979. 
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• Farley, Joanne. Vocational Education Outcomes: A Thesaurus oi Outcome Questions. 
Re!?earch and Development Series No. 170; Columbus: National Center for Research in 
Vocational Education, The Ohio State University, 1979. 

• Taylor. Carolyn M.; Darcy, Robert L.; and Bolland, Kathleen A. Vocational Education 
Outcomes: Annotated Bibliography of Related Literature. Bibliography Series No. 48. 
ColumbusJSJational Center for Research in Vocational Education, The Ohio State 
University, 1979. 

A related National Center project addressed the problem of interpreting outcome measures in 
vocational education (McKinney, Gray, and Ab ram; September 1978) and also resulted in the publica- ^ 
tion of a collection of papers expressing alternative viewpoints on outcome measures and their 
interpretation (McKinney and Harvey, September 1978). Two evaluation handbooks were produced 
dealing with vocational education outcomes (Franchak and Spirer; 1978 and 1979); and five state-of- 
the-art papers touching on various aspects of outcomes evaluation were published by the National 
Center (includin g Grasso, 1 979). A report on the status of vocational education in the United States as 
of 1975-76 was issued (National Center, 1978) and plans were made for publishing a similar report in 
1980 covering the 1978-77 school year and including some outcome data collected in early 1978. 



Phase Two of the ongoing study of vocational education outcomes carried the investigation 
to a more operational level. Including some preliminary empirical work, m effect, this phase has 
preparea a research, development, and evaluation (RD&E) agenda for conducting definitive 
empirical studies on a wide range of employment-related outcomes, as well as other types of 
outcomes, and for examining the correlates of particular outcomes, i.e., programmatic and 
nonprogrammatic factors significantly associated with observed outcomes. 

Related projects at the National Center during 1979 involved a preliminary investigation of 
the correlates of successful job placement of former vocational students, the preparation of 
handbooks on performance testing, and development of specifications for conducting 
longitudinal studies of former vocational students. 

Outside the National Center, a major study and evaluation of vocational education (including 
the effects of vocational programs) was under way at the National Institute of Education (David, 
1979). Evaluation studies on vocational education outcomes were reported by the Carnegie 
Council (Grasso and Shea, 1979), the National Bureau of Economic Research (Meyer and Wise, 
May 1979). and other sources. 



C. Organization of This Report 

In Chapter II, fifteen possible outcomes of vocational education are identified and -jiscussed 
in terms of their importance, appropriateness, and feasibility for use as evaluation criteria. 
Chapter III identifies fourteen "Essential Elements of Information for An Evaluation Study" and 
provides notes for operationalizing fifteen outcomes. Chapter IV gives a brief report on the pilot 
testing of one outcome, "Reducing the Risk of Unemployment for Minority Youth," carried out in 
collaboration with vocational education personnel in two different states. Chapter V summarizes 
findings and offers recommendations for further research, development and evaluation activities. 



2 



ERIC 



9 



11. OUTCOMES AS EVALUATION CRITERIA 



Some basic concepts related to outcomes evaluation, discussed in more detail elsewhere 
(Darcy. 1979 a.b). are briefly recapitulated In this chapter Then fifteen possible outcomes of 
vocational education are listed and discussed in terms of their Importance, appropriateness, and 
foaslbility for use In evaluation. 



A. Concepta and TeinilnolOQy 

In this study, vocational education outcomes are broadly defined as the consequences of 
vocational programs. Outcomes Include ^'products," "outputs." '^effects." 'Results,** and ''impacts.*' 
No technical distinctions are suggest d here for differentiating these various terms. Program 
consequences can usefully be descriljod in terms of (1) a time framework (e.g., immediate, 
short-term, and long-term outcomes). (2) the amount of Influence or control a program operator 
(e.g., a local education agency or local high school) exercises with respect to particular 
consequences, and (3) other considerations. It Is also useful to distinguish among possible 
outcomes, desired outcomes, and acfua' outcomes. Vocational education outcomes are 
manifested as changes in individual or societal capabilities, attitudes, status, or circumstances. 
The present report focuses on outconries in their role as evaluation criteria, i.e., the tests or bases 
in terms of which vocational programs are to be judged. 

Outcomes can be verbalized In the form of statements or questions. An example of an 
outcome statement is the declarative sentence; "Completers of high school vocational programs 
earn higher hourly wages three years after leaving school than comparable completers of college 
preparatory or general curricula." An outcome statement describes what happens to whom, as 
well as how, where, when, and why It happens. 

We can identify six elements of a complete outcome statement: (1) the change in capability 
or circumstance that constitutes the outcome; (2) an entity (person or thing affected by the 
program) in which the outcome is manifested; (3) the program or ''treatment" that induces the 
outcome; (4) the agency or channel through which the program is delivered; (5) a time 
framework; and (6) a rationale or theoretical explanation of why the outcome results from the 
program. 

In the example cited above, the outcome (i.e., hypothesized dependent variable) is higher 
hourly earnings. Program completers are the affected entity. High school vocational programs 
are the "treatment" (i.e.. strategic independent variable). The high school offering the program is 
the agency. Three years after leaving school is the time frame. Unstated above was a rationale 
linking higher earnings to participation In a vocational program (e.g.. Instruction in occupational 
skills helps inr iase productivity and qualifies workers for better paying jobs). Not every 
outcome statt lant will address all six elements of information; but as a minimum, a meaningful 
outcome statement will clearly identify the outcome, affected entity, and program "treatment." 

Outcomes, as suggested earlier can also be described in the form of questions. Example: 
"Do completers of high school vocational programs earn higher hourly wages three years after 
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leaving school than comparable completers of academic or general curricula?" Or the question 
may be phrased In more neutral terms; "How do the hourly earnings of high school vocational 
completers compare with the earnings of similar completers of academic or general curricula 
tliree years after jeaving school?" (The question "What are the rtioan hourly earnings of high 
school vocational completers four years after leaving school?" would be a possible research 
question or "follow-up" question but would not, in the viev.' of the project staff, constitute an 
evaluation inquiry since it provides no basis for an evaluative Judgment.) 

Because of two important advantages, the project staff elected to use the question format 
(Farley, 1979). First, it is easier to formulate outcome questions in more neutral terms than is the 
case with the outcome statements. Even more important, one is less likely to be misunderstood 
when a question mark rather than a period appears at the end of a sentence. Declarative 
sentences can give the Impression that the writer is asserting a claim or belief, or Indeed 
reporting documented facts regarding actual outcomes. At this stage of evaluative research in 
vocational education, however, there appears to be little documented knowledge that can be 
generalized with respect to actual outcomes. Questions concerning possible outcomes, on the 
other hand, abound. 



B. Some Key Outcomes of Vocational Education 

Table 1 lists fifteen possible outcomes of vocational education and corresponding questions 
intended to clarify the outcome titles. The list includes "job placement in training-related fields" 
(Outcome No. 14) and "satlsfactoriness to employer*^" (Outcome No, 11), two outcomes 
prescribed In the Education Amendments of 1976 as criteria for evaluating the effectiveness of 
vocational programs. Also included are consequences frequently cited as intended outcomes 
(i.e.. goals or objectives) of vocational education. Some others on the list are less familiar and 
may even suggest the possibility of unintended and detrimental effects. 

Several of the questions in Table 1 specify a limited program population (e.g.. Outcome No. 
3), a particular level of schooling (e.g.. Outcome No. 4) or a certain time framework (e.g.. 
Outcome No. 8). It should be understood that outcome questions can be written in a variety of 
ways, almost without limit. The specificity expressed in some questions appearing in this table is 
intended to demonstrate various possible evaluation studies, some of which will have greater 
relevance to a particular state or local school than broad-based studies or evaluations that focus 
on differen* subpopulations. There is nothing sacrosanct about any particular focus indicated by 
the questions as formulated in Table 1. 

This list of outcomes.does not claim to be authoritative, much less authoritarian. It is best 
perceived as a sample of the numerous and diverse possible outcomes for which vocational 
education might be held accountable. These are "key outcomes" in the sense that they have 
been Identified by the project staff as being significant in themselves and as having special 
importance in evaluation, e.g., to Illustrate diversity, conceptual complexity, mothodologicai 
alternatives, and statistical problems. Which of these outcomes, if any, should be used for 
evaluation purpos^is in a given situation would presumably be determined by policy makers, 
evaluation sponsors, or program operators in particular state and local settings. 

This list of fifteen outcome questions can be compared with a list of thirty (Darcy, 1979a) 
and the extensive, categorized list of 252 (Farley, 1979) that appeared in earlier project reports. A 
list of ten outcome statements was used as the basis for discussions by a national panel (Darcy, 
Bolland, and Farley, 1979). And a list of twenty outcome questions was included in a paper 
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Table 1 

SOME KEY OUTCOMES OF VOCATIONAL EDUCATION 


Outcome 


Outcome Question 


VI; improving oasic eaucaiionai 
skills ' 

1 

/ 


To what extent do students who are enroiled in 
vocational education improve their basic educa- 
tional skills (com.munications and numerical 
calculation)? 


(2) Development of useful 
• occupational skills 


Do students acquire useful occupational skills 
by participating in vocational programs? 


(3) Reducing the r^sk. 
of unemployment 


What effect does completion of a high school 
vocational program have on the risk of unem- 
ployment for minority youth? 


^H/ MU^uiring WOria-OT-WOrK 

knowledge 


Do postsecondary vocational students acquire 
greater world-of-work knowledge from their 
programs than comparable nonvocational stutc 
dants derive from theirs? (Defined to include 
occupational information and knovyledge of 
. labor market and work adjustment processes.) 


(5) Effect on educational 
commitment 


Does enrollment in a high school vocational 
program strengthen the educational commit- 
ment of economically disadvantaged students? 
(As measur J by school attendance records and 
dropout ri:ves.) 

♦ 


(6) Development of leadership 
qualities 


Does enroMmenJ in a high school vocational 
program promote the development of leadership 
qualities? 


(7) Postsecondary educational 
progress 


What effect does enrollment in a high school 
vocational program have on access to and early 
success in various forms of postsecondary 
education'? 


5 



Table 1 (continued) 
SOME KEY OUTCOMES OF VOCATIONAL EDUCATION 


Outcome 


Outcome Question 


(8) Level of postschool earnings 


What effect does vocational education have on 
the earnings of young workers during their first 
two years after leaving school? 


(9) Satisfaction with current 
school experience 


Do students enrolled in vocational programs 
express more. less, or the same satisfaction with 
their current school experience as comparable 
nonvocational students? 


(10) Job-search time 

/ 


What effect does vocational education have on 
.the length of time it takes young workers to frnd 
Tuii-tirne employment aner leaving nign sLriuuir 


(II) Satisfactoriness to employers 


How do employers rate former vocational stu- 
dents as compared with comparable nonvoca- 
tional students in terms of attitudes, abilities, 
and performance on the job? 


(12) Attractiveness of the 

community for industrial 
development 


• 

What effect does vocational education have on 
the community as a site for industrial location or 

expansion? 

> 


(13) Employment opportunities 
for minority .workers 


What effect does the existence of an extensive 
vocational education program have on the 
occupational distribution of minority workers? 


(14) Job placement in 

training-related fields 


To what extent do former vocational students 
find employment in occupations related to their 
training? 


( 1 Q) Dovolopment of self- 
help skills 


To what extent do high school vocational 
students acquire economically valuable consu- 
mer and other self-help skills? 



SOURCE. Proiect staff, National Center for Research in Vocational Education, The Ohio State University, 1979. 
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presented at the 1979 annual'iCfonferonce of the American Vocational Education Research 
Association (Darcy, 1979b). The latter consisted of the fifteen questions appearing in Table 1 
plus the following: \ 



Outcome 



• Effect on student self-image 



• Economic returns 



Outcome uuestion 



Does vocational oducation have.a more favora- 
ble or less favorable effect on student self- 
confidence and self-imageUba<K^ 
general/academic curricula? ^ 

Are the economic returns on investments in 
vocational education lower, higher, or the same 
^ returns on investments made in academic/ 
general programs below the baccalaureate 
Jevel? 



• Socioeconomic understanding 



• Effect on early job experience What effect does enrollment in a vocational 

curriculum have on the quantity and quality of 
jQb experience that young people acquire before 
they leave high school? , ^ 

Do students who complete vocational programs 
develop a better understanding of the practical 
realities of our economic system than students 
• - who complete the same number of years of 
schooling in a general/academic program? 

Do completers of vocational programs at the 
postsecondary level ekpress higher, lower, or 
the same level of job satisfaction when com- 
pared with similar young people having the 
same quantity of schooling, after five years of 
post-school employment? 

Some evaluation experts with whom the project staff consulted have suggested additional 
outcomes for inclusion in a basic list, notably "employability development." This one. among 
others, was omitted because it was considered too resfstant to ope^rational formulation. 



• Job satisfaction 



C« Conolderattons Influencing the Choice of Outcome Criteria 

To be suitable for use as an evaluation criterion, an outcome must be important and 
appropriate: i.e.. not trivial and not simply of marginal relevance to the mission of vocational 
education. The outcome must also be feasible to operationalize in the sense that, unlike the 
question of how many angels can dance on the head of a pin, it can actually be studied 
empirically. Feasibility can be defined to include issues of cost, timing, administrative burden, 
information access, propriety, acceptability^ and the likelihood that people will be cooperative in 
conducting the study. Finally, an outcome selected for study should hold promise of generoting 
information that can be used in making decisions that will actually influence programs or 
policies. 



The relative importance of the respective considerations will vary with circumstances, but it 
seems clear that hypothesized outcomes should meet minimum standards with respect to each of 
these sets of considerations. Moreover, one must resist the temptation to study what is easy to 
study rather than what ought .to be studied (Weiss, 1972). At the same time evaluators should be 
encouraged to express concerns they may have about the practical difficulties they anticipate in 
studies they are asked to conduct. 

. In the judgment of the project staff, many outcome criteria fail to qualify as suitable on the 
basis of the above considerations. Informal discussions with members of the vocational 
education community and with experienced program evaluators (see Section D below) indicate 
widespread agreement with this perception. Even familiar evaluation criteria, such as placement 
in training-related jobs and "satisfactoriness" to employers may have doubtful merit in terms of 
importance/appropriateness, feasibility, and usefulness for decisionmaking. Yet such criteria are 
mandated as the basis for evaluating program effectiveness (Education Amendments of 1976, 
Sec 112b). 

Choosing evaluation criteria is a topic that has evidently received little professional attention. 
Thus, there not only appears to be a lack of agreement on which particular outcomes should be 
used for evaluating vocational education (Reubens 1974, and Section D below) but also no 
consensus on procedures that could help in the selection of an accepted set of outcomes. 
Extensive discussion of this point will not be presented here, but it is suggested that choosing 
evaluation criteria is a critical step in any evaluation study. More attention should be paid to the 
rationale for such choices, i.e., to the cogency of the arguments in support of these instrumental 
value judgments. 



D. The "Validation" of Outcome Criieria 

Valid criteria for appraising vocational education on the basis of outcomes may be thought 
of as criteria that do what Ihey purport to do. Should vocational education be held accountable 
for the success or failure of former vocational students in the job market? for the sort of citizens 
they become? for whether they graduate from high school? Value judgments are called for in 
establishing evaluation criteria (as well as setting evaluation standards). The task is similar to. 
though not precisely the same as, defining the mission and purpose of vocational education. ' 

There is no simple way to "validate" outcome criteria apart from a simplistic appeal to 
authority (whether by legislative mandate or opinion surveys). This .is not to suggest that one 
criterion ib as good as another but rather to recognize that thoughtful inquiry, empirical data, 
and well-informed dialogue are required to generate the right questions to be addressed by 
evaluative research. These questions are likely to be numerous, to vary among states and local 
communities, and to change with the pa.ssage of time. 

After considerable conceptual exploration and a limited amount of mostly unsatisfactory 
empirical investigation, the staff decided to poll a small, nonrandom jample of iridividuals 
knowledgeable in Vv-cational education and experienced in program evaluatiorr Nine experts, 
responding individually and informally (not by means of a Delphi or Nominal Group Technique), 
were asked to rate the fifteen outcomes listed in Table I on the basis of (a) appropriateness/im- 
portance, (b) feasibility, and (c) overall rating, all things considered. Table 2 shows the results. 
Attachment A reproduces the instructions given to the raters, while Attachment B identifies the 
nine individuals who submitted the ratings. 



Checkmarks in the extreme right column identify the outcomes that received high overall 
ratings from five or more of the nine panelists. The one unanimous choice was "Developing 
Usoful Occupational Skills" (No. 2). But iRven here, three of the raters expressed reservations 
about the feasibility of evaluating vocational education on the basis of this criterion. 

Two other outcomes received five high ratings; "Improving Basic Educational Skills (No. 1} 
and "Effect on Educational Commitment" (No. 5). 

The remaining twelve outcomes— including the two mandated in the Education Amendments 
of 1976 (No. 11 and No. 14)— failed to generate much enthusiasm from the group. 

No claim is made that these ratings are representative of any particular large group. Based 
on the exploratory work alluded to above, as well as on statements reported in the professional 
and popular media, we suspect that diverse constituencies would express conflicting views. One 
may infer that no broad consensus exists concerning the particular outcomes that should be 
used as criteria for evaluating vocational education. Specific states, local communities, and 
individual schools may attach great importance to different outcomes at different times, in 
different situations, and argue-convincingly that these are the outcomes for which they choose to 
be accountable. 

One implication of nonconsensus— as illustrated in Table 2— may be that far more thought, 
data, and dialogue are needed as a basis for determining the mission of vocational education and 
the particular outcomes for which it should realistically be held accountable. An outcome 
"validation" exeicise involving education officials, advisory councils, policy analysts, and 
experienced evaluators could stimulate revealing find fruitful discussions at community, state, 
and national levels. 
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TABLE 2 

RATINGS OF FIFTEEN POS'?IBLE OUTCOMES OF VOCATIONAL EDUCATION 
FOR USE AS EVALUATION CRITERIA (N=9) 



Outcome & Outcome Question 


Appropriateness 
& Importance 


Feasibility 


Overall 
Ratiig 


High Ovfirall 
Ri'ting 


1. IMPROVING BASIC EDUCATIONAL SKILLS. 
How effectively do students who are enrolled in 
vocational education learn the basic education 
skills of commun-'^ation and numerical calculation? 


High -5 
Interm. —2 
Low -2 


High -5 
Interm. -4 
Low -0 


High 

Interm. —2 
Low —2 


✓ 


2. DEVELOPING USEFUL OCCUPATIONAL SKILLS. 
Oo students acquire useful occupational skills by 
vocational programs? 


High ^9 
Internr. —0 
Low -0 


High ^6 
Interm. —3 
Low —0 


High -9 
Interm. —0 
Low —0 




?. REDUCING THE RISK OF UNEMPLOYMENT. 
What effect does completion of a high school 
vocational program have on the risk of 
unemplovment for teenage minority workers? 


High * -4 
Interm. —2 
Low -3 


High -2 
Interm. —4 
Low -3 


High -1 
Interm. —5 
Low -3 




4. ACQUIRING WORLD-OFAVORK KNOWLEDGE. 
Do postrecondary vocational students acquire 
greater world-of-work knovi/ledgfi than comparable 
nonvocational students? (Defined to include 
occupational information and knowledge o.' labor 
market pr . .ses.) 


High -5 
Interm. —4 
Low -0 


High -4 
Interm. —5 
Low -0 


High -2 
Interm. —7 
Low 0 




5. EFFECT u. , EDUCATIONAL COMMITMENT. 
Does enrollment in a high school vocational 
program strengthen the educational commitment 
of economicall^tlisadvantaged students^ (As measured 
by school attendance records and dropout rates.) 


Hiqh -5 
Interm. —3 
Low - 1 


High -7 
Interm. —0 
Low -2 


High -5 
Interm. -2 
Low -2 




6. DEVELOPMENT OF LEADERSHIP QUALITIES. 
Does enrollment in a high school vocational pruc,* 
promote the development of leadership qualities? 


HiQh -1 
interm. —6 
Low -2 


High ^0 
Interm. —3 
Low —6 


High -0 
Interm. —5 
Low -4 




7. POSTSECONDARY EDUCATIONAL PROGRESS. 

wVh.Tt pffppt Hooc pr%rnll rT\pnt In 'i hInH cnKriol vr^/^nt Ir^n ^sl 
vviiiii LJLiUd c tiullMtClll III u iiiyii sCfivJUl ^OCallOnni 

program have on access to and early success -n various 
forms of postsccondary education? 


High -1 
Interm. —7 
Low -1 


High -4 
Interm. —4 
Low -1 


High -1 
Interm. —6 
Low -2 




8. LEVEL OF POSTSCHOOL EARNINGS 

What effect does vocational education have on 
earnings of young workers during their first 
two years after leaving school? 


High -5 
Interm. —3 
Low -1 


High ^3 
Interm. —5 
Low -1 


High -4 
Interm. -4 
Low -1 






TABLE 2 (continued) 



Outcome & Outcome Question 


Appropriateness 
Hi Importance 


Feasibility 


Overall 
Hating 


High Overall 
Rating 


9. SATISFACTION WITH CURRENT SCHOOL EXPERIENCE. 
Do students who are enrolled in vocational programs express 
morft, less, or the same satisfaction with their current school 
exparience as comparable nonvocational students? 


High -4 
Interm. -4 
Low —1 


High -4 
Interm. -4 
Low -1 


High ^1 
Interm. —8 
Low -0 




10. JOB-SEARCH TIME. 

What effect does vocational education have on the length 
of time it takes young workers to find full-time employ- 
ment after leaving high school? 


High -4 
\lnterm. —4 
Low —1 


High -3 
Interm. -5 
Low -1 


High ^2 
Interm. —7 
Low -0 




1 1 . SATISFACTOR INESS TO EMPLOYERS, 

How do employers rate former vocational students as 
compared with comparable nonvocational students in 
terms of attitudes, abilities, and performance on the job? 


High- -7 
Interm'. —1 
Low —1 


High -1 
Interm. —5 
Low —3 


High -4 
Interm. —4 
Low —1 




12. ATTRACTIVENESS OF THE COMMUNITY FOR INDUSTRIAL 
DEVELOPMENT. 
What effect Cf^as vocational education have on the attractiveness 
of the community as a site for industrial location or expansion? 


High -2 
Interm. -6 
Low —1 


High -0 
Interm. -1 
Low -8 


High -1 
Interm. —4 
Low —4 




13 EMPLOYMENT OPPORTUNITIES FOR MINORITY WORKERS. 
What effect does the existence of an extensive vocational 
fiducation program have on the occupational distribution of 
minority workers? 


High 

Interm. -3 
L<ow -2 


High ^1 
Interm. —1 
Low -7 


High -1 
Interm. —6 
Low —2 




14. JOB PLACEMENT IN TRAINING-RELATED FIELDS. 

To what extent do former vocational students find employment 
in oixupations related to their training? 
— 


High -6 
Interm. —4 
Low -0 


High --3 
Interm. -6 
Low -0 


High .-4 
Interm. —5 
Low -0 




15. DEVELOPMENT OF SELF-HELP SKILLS. 

To what extent to high school vocational students acquire 
economically valuable consumtr and other self-help skills? 


High -4 
Interm. —4 
Low —1 


High -1 
Interm. ~3 
Low —5 


High -2 
Interm. —5 
Low -2 





SOURCE. See text and Attachnneru A and B. National Center fpr Research in Vocational Education, The Ohio State University, 1 979. 
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Table 2 (continued) 
ATTACHMENT A 

Instructions for Rating Outcome Criteria 



Listed on the following pages are fifteen possible outcomes.or consequences of vocational 
education along with questions intended to define the respective outcomes in more specific 
terms. (These outcomes were selected by the project staff from a file of over 200 possible 
outcomes as being significant, though not necessarily the most important of all outcomes that 
might be selected for use as evaluation criteria. They are not intended to represent the goals or 
objectives of vocational education.) ' 

Please use the terms High, Intermediate, or Low (write "High." '^Intermediate/' or "Low" in the 
respective columns) to rate the outcomes exactly as they are described, without changing the 
wording or indicating any particular Interpretations or reservations you might have. (We 
recognize that many of the outcome questions could be replirased to apply to different 
populations such as secondary rather than postsecondary. or all students rather than just 
minority students,) 

"Appropriateness and Importance" refers to the suitability of using the particular outcome as one 
of several criteria for evaluating vocational education, on the assumption that it can be 
operatlonalized, A rating of High in the column headed "Appropriateness and Importance" means 
you feel that particular outcome is highly appropriate and important. A rating of Low means that 
it is not appropriate for vocational education to be evaluated on the basis of the outcome, even 
though the outcome may be important for some other program. 

Feasibility refers to the practicality of using the particular outcome a3 an evaluation criterion. 
Given the various conceptual, statistical, administrative, legal, and other problems, how feasible 
is it to use the outcome? ''Feasible" means the evaluation i^ capable of being accomplished— not 
whether it should "theoretically" be done, but whether it csn be done. 

An Intermediate rating in the "Feasibility" colurpn for a given outcome indicates that it would be 
quite possible to collect valid ddta and carry out the evaluation, though not without difficulty or 
substantial cost in terms of administrative burden, lapse of time, expense, etc. 

An "OveraH Rating" of Low indicates that on balance— considering both appropriateness/impor- 
tance and feasibility— you would not recommend that this outcome be used as one of half dozen 
or so outcome criteria on which the success or failure of vocational education might be judged. 
A High rating means all things considered you would recommend its use. The overall rating will 
approximate a weighted average of appropriateness/importance pius feasibility, with you 
providing the implicit weights. 
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Table 2 (continued) 

ATTACHMENT B 

Individuals Who Rated the Fifteen Outcomes 

DR. HERMINE CHERN. Associate in Charge, Career Education Evaluation Unit. School District 
of Philadelphia, Philadelphia, Pennsylvania. 

Conducts evaluation studies of vocational education and career education programs involving 
inner-city youth and adults; member of American Educational Research Association, Pennsylva- 
nia Educational Research Association. American Personnel Guidance Association; author of 
Evaluation of Federally Funded Vocational Education Projects, 1979. 

DR. JESSE S. CLEMMONS. Director, Program Improvement Unit, Division of Vocational 
Education. State Department of Public Instruction, Raleigh, North Carolina. 
Former high school vocational teacher; more than ten years of research and evaluation 
experience in vocational education; responsible for management information system, research 
coordinating unit, and statewide evaluation programs in North Carolina; doctoral studies at 
Pennsylvania State University emphasized evaluation and research; active in National Conference 
of RCUs. 

DR. GEORGE H. COPA. Director, Minnesota Research and Development Center, Department of 
Vocational and Technical Education, University of Minnesota, Minneapolis, Minnesota. 
Professor of vocational and technical education; past president of the American Vocational 
Education Research Association; member of the National Center's Evaluation Technical Advisory 
Panel. 

DR. JOHN T. GRASSO. Research Associate, Office of Research and Development, Center for 
Extension and Continuing Education, West Virginia University, Morgantown, West Virginia. 
Co-author of Vocational Education and Training: Impact on Youth, 1979, and other publications 
primarily based on analysis of data from the National Longitudinal Surveys of young men and 
women conducted by Ohio State University and the U.S. Department of Labor; holds a faculty 
appointment in College of Human Resources and Education." West Virginia University. 

MRS. GUADALUPE MCDOUGALD. Assistant to the President, Harry S. Truman College, 
Chicago. Illinois. 

Member of Illinois State Advisory Council on Vocational Education; experienced in conducting 
educational evaluations at secondary and postsecondary levels; formerly on administrative staff 
of Chicago Public Schools; member of National Association for Bilingual and Bicultural 
Education; past president of Adelante, statewide education organization; Outstanding Young 
Women of America award. 1978. 

OR. DOUGLAS PATTERSON. Director. Research Coordinaling Unit, Division of Vocational 
Education Services. State Department of Education. Montgomery. Alabama. 
Has directed vocational program evaluations at state and local levels in four different states 
during the past ten years; active member of the American Vocational Education Research 
Association and National Conferenco of RCUs; past president of Southwide Research 
Coordinating Council and Southern Region Research Contere..ce in Agricultural Education. 
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'DR. DOUGLAS SJOGREN. Professor of Education, College of Professional Studies, Colorado 
State University, Fort Coliins, Colorado. 

Teacher of graduate and undergraduate courses in statistical methods, research, evaluation, and 
educational growth and development; author of publications in occupational analysis, statistics, 
and educational evaluation and research; consultant and director of evaluation studies at 
secondary and postsecondary levels. 

DR. WILLIAM W. STEVENSON. Senior Research Specialist, National Center for Research in 
Vocational Education, The Ohio State University, Columbus, Ohio. 

Project director, evaluation technical assistance to selected states; former high school vocational 
teacher and head of the Division of Research, Planning, and Evaluation, Oklahoma State 
Department of Vocational Education; author of Vocational Education Evaluation: Problems, 
Alternatives, and Recommendations, 1979; past president of the National Association of RCU 
Directors. 

OR. JERRY P. WALKER. Senior Research Specialist, National Center for Research in Vocational 
Education, The Ohio State University, Columbus, Ohio. 

Has conducted evaluations of educational and other human service programs; Associate Director 
of the National Center's Evaluation Division for seven years; active member of American 
Educational Research Association; consultant in evaluation, policy analysis, and institutional 
planning. 
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III. EVALUATION PROCEDURES 



This chapter identifies fourteen elements of information considered by the project staff to bs 
essential for evaluating vocational education on the basis of outcomes. Following a brief 
discussion of three of the most crucial elements-empirical Indicators, evaluation design, and 
evaluation standards— notes are provided for use in operationalizing each of the fifteen key 
outcomes identified in Chapter II above. 

A. EMtntlal Elements of Information 

Table 3 lists fourteen items of information considered by the project staff to be essential for 
planning, carrying out, and reporting evaluation studies focusing on vocational education 
outcomes. The proced*jic»s die based on a conceptual model according to which a program (e.g., 
vocational ^ 'ucation) is judged ''good'* or ''bad" in terms of a specified critorion (e.g., an 
hypothesized outcome selected as the test or basis for judging merit) to the extent that a given 
empirical indicator (e.g., a test score, unemployment rate, etc.) attains a specified evaluation 
standard (i.e., relative or absolute level of performance). Thus, in program evaluation there must 
always be a criterion (test), an empirical indicator (measure), and an evaluation standard 
(specified performance level). 

Any resulting evaluative judgment would be limited and tentative, based on the particular 
outcome criterion and data observations. No comprehensive, definitive judgment concerning the 
overall merit of a program should be formulated until a broad range of appropriate criteria are 
examined and sufficient empirical observations made both for the program population and the 
comparison group. The chief virtues claimed for the procedures listed in Table 3 are explicitness, 
comprehensiveness, and provision for reporting and applying evaluation findings. 

The meaning and Importance of each element in the checklist is perhaps beUer conveyed 
through specific illustrations than with general definitions. Table 4 applies the checklist to all 
fifteen of the key outcomes identified in Chapter 11. Only brief explanations are provided here for 
the respective elements. 

Checklist Item No. 1 requires a clear statement of the outcome question to be answered. 
This defines the problem to be studied, which is the first step in systematic decision-making 
(Darcy and Powell, 1973). Item 2 identifies the particular change that is hypothesized to result 
from exposure to the independent variable (though not yet the specific way of measuring that 
change). Item 3 describes the student population or other entity (such as a school, local labor 
market, or the community as a whole) in which the outcome is hypothesized to have occurred. 
Item 4 describes the program or "treatment" that is identified as the (most important) 
indupendent variable in the study. Item 5 provides a theoretical explanation of why there may 
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Table 3 

ESSENTIAL ELEMENTS OF INFORMATION IN AN OUTCOMES EVALUATION 





CHECKLIST OF ELEMENTS 


(1) 


Outcome question to be answered. 


(2) 


1 Outcome hypothesized as the dependent variable. 




Affected entity (student population or other entity in which the outcome Is 
observed). 


(4) 


Program identified as the strategic independent variable. 


\0) 


Hationale for the hypothesized outcome and appropriateness of the outcome 
as a basis for evaluating the program. 


(C) 


Empirical indicators of the outcome. 


I // 


Methodological considerations (evaluation design, sampling procedures, 
statistical analysis). 


(8) 


Evaluation standard (basis for interpreting outcome data to judge program 
effectiveness). 


(9) 


uaia requiremenTs imstruments, procedures, data base). 


(10) 


Feasibility of investigating this outcome (conceptual, administrative, cost, 
time, other considerations). 


(11) 


Findings (results of data analysis, warranted inferences, generalizability). 


(12) 


Potential impact of findings (implications for policy, program design, and 
management, image of vocational education). 


(13) 


Dissemination of findings and suggested applications.) 


(14) 


Implications of findings for further research, development, and evaluation 
(RD&E) activities and relationship of this evaluation to previous studies. 



SOUHCfc": Pro)oct staff, National Center for Research in Vocational Education, The Ohio State University, 1979. 
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indeed be a significant association or linkage between the independent and dependent variables. 
Th!s Item also calls for a rationale to justify using the speciftdd outcomes as a basis (perhaps 
one among several) for appraising the efficacy of the Independent variable. 

Item 6 identifies one or more specific ways of measuring the. outcome in question. Item 7 
indicates how the evaluation study is to be approached methodological!/. Item 8 posits the level 
of attainment to which the performance of the affected entity will be compared. And Item 9 
specifies the kinds of information required to generate the necessary empirical indicators, along 
with data-collection instruments and procedures that may be appropriate. Because Items 6, 7, 8, 
and 9 are so crucial for a bona fide, data-based evaluation, additional attention is given to them 
-in^Sectlo .■ B and C below. ^ 

Checklist Item 10 is intended to alert the designers of the evaluation study to potential 
hazards, costs, and limitations. Items 11, 12, 13, and 14 are both futuristic and retrospective, 
inviting evaluators first to consider how they might deal with different possible results, 
(anticipation of which might induce timely modifications In how the study is finally designed), 
and then what to do with the actual results once they become known. A major question 
addressed by Item 11 concerns validity, both Internal dnd external (Anderson et al., 1975). Facts 
seldom speak for themselves. What inferences are warranted from the facts observed in the 
evaluation study in question? And to what extent can the Inferences be generalized to other 
populations? Item 12 relates to the utility of the study, reminding one that the purpose of 
evaluation, in contrast to research, Is not to produce knowledge for its own sake but rather to 
generate information that will be useful in making decisions about the program that wae the 
object of the evaluation study (Getting, 1976 Part I). Item 13 Is the proc jss that can transform the 
potential Identified In Item 12 into practical applications and actual benefits of the evaluation 
study. Finally, Item 14 calls for the evaluator to relate the procedures and findings of the study to 
those of past investigations and to suggest future lines of inquiry that might be especially vrultful 
in explaining what works, how, and why In this particular area of vocational education. 



B. Empirical Indicators of Program Outcomes 

One of the information elements that Is of crucial importance in p'^ evaluative outcome study 
is data on program results. Getting and Cole (1978) identify faulty c come measurement as the 
most likely source of failure in any program evaluation. They further remind us that you cannot 
make up for poor execution at one step in the evaluation process (e.g., data collection) by 
excellence in another (e.g., data analysis). The steps are multiplicative; the quality of an 
evaluation study is the product of interaction among ail the parts. 

In vocational education evaluation, a number of data problems exist. Some of these were 
identified and illustrated in earlier reports (Darcy, 1979a; Bolland, 1979). During the past year, 
the National Center published two state-of-the-art papers dealing specifically with data Issues. 
Hopkins (1979) describes vocational education data needs, types, and sources, listing as primary 
sources of evaluative data the U.S. Bureau of Labor Statistics, State Employment Security 
Agencies, and State Departments of Education. Lee (1979) raises questions about the extent to 
which vocational education administrators use evaluative data and identifies a number of 
conuitions that must be met before such data will In fact be used. A third state-of-the-art paper 
(Grasso, 1979) alludes to shortcomings of data on training-related placement. The National 
Center's report on the status of vocational education (1978) cJoais at length with data problems 
and prospects for improvements in the data base. 
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In the present study, the staff was committed to using only existing data.- Although questions 
were raised about methods of collection, reporting, and state-level analysis, the scope of work 
did not include intensive probing of state data systems or field audits (Darcy, 1979b). The staff 
wishes to emphasize, however that without adequate and accurate data no valid empirical study 
Of vocational education outcomes is possible. 

C. Standards for Interpreting Outcome Data 

The issue of evaluation standards was addressed in an earlier report (Darcy, 1979a). Efforts 
have been made to underscore the nature of evaluation standards and their role in conducting 
bona fide evaluation studies as opposed to simple descriptive reports, the meaning of "standard" 
in the context of educational evaluation, as pointed ogt by Anderson et al. (1975). conforms to a 
.dictionary definition: "a degree or level of requirement, excellence, or attainment." The concept 
can be illustrated with an analogy. 

In evaluating the performance of an automobile, one might posit the criterion of fuel 
efficiency as indicated by the number of miles the car can be driven per gallon of gasoline 
consumed. Assume that the designated automobile Is obsen/ed to average twenty-four miles per 
gallon in freeway driving. Whether this performance is good. bad. or indifferent depends on the 
standard that is adopted for comparison, e.g... 20.5 miles per gallon. In practice, the search for 
suc> a standard might be frustrated by the nonavailability of a car of equal size, weight, and 
price, or simply by the lack of documented fuel consumption for the comparison car. 

In^the notes for operationaiizing outcomes appearing in Section D. three different evaluation 
designs are utilized (see entries for Item 7 in "Essential Elements of Information"). Some possible 
evaluation standards are then suggested (Item 8 entries) that correspond to the respective 
designs. Various d^ta collection methods are also mentioned (Item 9). 

The thrp- designs are (A) Comparison Groups, (3) Before and After, and (C) Case Study 
Design A resembles the "nonequivalent control group design" described by Campbell and 
Stanley (1963). All three designs are described in Borus (1979). In some Inquiries there is really 
no evaluation study at all because no basis for evaluation exists other than intuition or arbitrary 
judgment (Clarkson. Neuberger, and Koroloff. 1977). Differences among experimental, quasi- 
experimental, and other research designs are discussed in. the books by Campbell and Stanley 
and by Borus cited above. Covarlance analysis and multiple regression techniques used to control 
statisticallv for preexisting differences are described in Grasso (1975). 



Many of the entries for Item 8 suggest that the performance of vocational students be 
compared with that of nonvocational students (i.e.. college preparatory and general) who have 
similar charactenstics. A good deal of discretion and ingenuity is called for In conjuring up 
appropriate comparison groups or. instead, developing other types of evaluation standards that 
make comparison groups unnecessary. One innovative approach to the development of 
quantitative evaluation standards— exemplified by the unemployment disparity index-is 
described in Chapter IV 



t>. Nol«8 for Operationaiizing Fifteen Outcomes 

Table 4 provides notes for use in operationaiizing fifteen possible outcomes of vocational 
education. Those aro the sanio outcomes discussed in Chapter II. As indicated earlier, these 
outcomtjs are not purported to be the most important outcomes of vocational education, nor are 
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they necessarily the fifteen most appropriate or feasible outcomes to use for evaluation 
purposes. They are outcomes the project staff considers to be significant in themselves and 
useful for the dual purpose of illustrating (a) diverse types of possible outcomes and (b) different 
methodological approaches to evaluating vocational education on the basis of outcomes. The 
''validity" of the particular outcome criteria included in this list was addressed in Chapter II. 

It is hoped that those notes will be genuinely useful to evaluators in state and local 
education agencies as well as suggestive to others interested in outcomes evaluation. We do not. 
however, pretend that they are detailed enough to take into the field. Many additional hours 
would be needed to design an evaluation study for any given outcome so that the study would fit 
a particular time, place, and requirements of the sponsoring agency. 

In order to avoid having a table of excessive length and to provide a consistent format that is 
relatively easy to follow, we have limited the entries to two pages per outcome. For each 
outcome, information is provided that addresses all fourteen items in the checklist "Essential 
Elements of Information in an Outcomes Evaluation.'* The entries in the column headed ''Information 
for this Outcome" are intended to (a) clarify the outcome to be studied, and (b) suggest possible 
ways to operationalize the outcome for evaluation purposes. Brevity prevails over comprehen- 
siveness in suggesting how to approach the development of an evaluation plan. 

For each checklist item, there are numerous potential alternatives. For example, Outcome 
No. 5 identifies the affected entity (checklist Item #3) as economically disadvantaged high school 
students. A particular local education agency (LEA) might want to know what effect vocational 
education has on all students, disadvantaged and nondisadvantaned alike. The student 
population can be readily changed to address the more Inclusive study. Similarly, Outcome No. 8 
refers to the earnings of former vocational students during their first two years after leaving 
school. If the evaluator is interested in determining earnings over a three-year or four-year time 
period, this modification can be made. Indeed, a fifty-page planning paper could be written on 
each outcome identifying possible variations in such study elements as the affected entity, 
program specified r.s the strategic independent variable, rationale for the hypothesized outcome, 
empirical indicators, data requirements, time framework, evaluation standards, etc. 

The specificity expressed in some of the notes is intended to suggest that evaluation studies 
require particular definitions and delimitations of some kind in order for the evaluation plan to be 
operational. What is required is a rifle approach, not a shotgun approach. But there are many 
possible targets. 

While Table 4 does not fully operationalize the fifteen outcomes, it does carry the process a 
step or two beyond identifying outcomes and simply listing a set of general evaluation 
procedures. The task of designing a completely operationalized evaluation study, however, is left 
to state and local education agencies or other evaluators based on their particular interests, 
resources, and needs. 

A list of abbreviations used in the table appears at the end of the table. In a few cases 
material is carried over into the end notes, and for each outcome two or. more literature 
references are provided. The notes are designated by numbers (corresponding to the number of 

the outcome) and by letter ("a" for the first note applying to a particular outcome, "b" fo» the 
second, and so on) 
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Table 4 ^ 
SOME KEY OUTCOMES FOR USE IN EVALUATING 
VOCATIONAL EDUCATION PROGRAMS 

(Table begins on following page) 
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Table 4. SOME KEY OUTCOMES FOR USE IN EVALUATING 



OUTCOME NO. 1 


• IMPROVING BASIC EDUCATIONAL SKILLS 


ESSENTIAL ELEMENTS OF INFORMATION FOR AN EVALUATION STUDY 




Checklist of Elements 


Information for This Outcome* 


(1) 


Out(x;^me question to be 
answered: 


To what extent do students enrolled in 
vocational programs improve their basic 
educational skills (communications and 
numerical calculation)? 


(2) 


Outcome hypothesized as 
the dependent variable: 


Improvement of basic educational skills. 


(3) 


Affected entity (student 
population or other entity 
in which the outcome 
is observed): 


High school students enrolled in vocational 
programs at the 10th, 11th, and 12th grade 
levels. 


(4) 


Program identified as the 
strategic independent variable: 


Enrollment in an approved vocational program 
at the lOtK, 11th, or 12th grade leval for a 

minimum of two semesters. 

^ _ g< 


(5) 


Rationale for hypothesized 
outcome and appropriateness 
of the outcome as a basis 
for evaluating the program: 


The practical orientation of vocational *udies 
may make it easier for students with certain 
characteristics to learn basic educational skills. 
Students may be more highly motivated to learn 
in general because of their interest in vocational 
subjects. 


(6) 


Empirical indicators of 
the outcome: 


Scores on verbal and quantitative tests, 


(7) . 


Methodological considerations 
(evaluation design, sampling 
procedures, statistical analysis): 

< 


Comparison-group design with pre- and post-tests; 
stratified random samples of vocational and other 
students (sex, race, etc.); statistical tests of the 
significance of differences. 


(8) 


Evaluation standard (basis for 
interpreting outcome data to 
judge program ^iffectiveness): 


Superior performance of vocational students vs. 
nonvocational students matched as nearly as 
possible for sex, race/ethnicity, mental ability, 
socioeconomic background, and quantity of 
schooling. 


(9) 


Data requirements (instru- 
ments, procedures, data base): 


Background data and test scores for samples of 
program population and comparison ^roup. 
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VOCATIONAL EDUCATION PROGRAMS 



Outcome No. 1 





Checklist of Elements 


Information for This Outcome* 


(10) 


Feasibility of investigating 
this outcome (conceptual, 
administrative, cost, time, 
other considerations): 


Potentially this outcome Is highly feasible sinbe 
the required data are routinely collected by most 
LEAs for all students. Additional costs for data 
analysis should be low. Administrative coopera- 
tion woulc be required between vocational and 
nonvocational departments. Some vocational 
officials could be reluctant to cooperate in the 
study because they feel that the outcome is not 
an appropriate evaluation criterion for vocational 
education. 


(11) 


Findings (results of data 
analysis, warranted infer- 
ences, general izability): 


If students having certain characteristics show 
greater basic educational improvement in a 
vocational curriculum than in an academic or 
general program, this fact would stimulate policy 
makers to re-think the role of vocational educa- 
tion in public education. It would be important 
to determine whether differential effects are 
observed by vocational program areas. 


(12) 


Potential impact of findings 
(implications for policy, 
program design and manage- 
ment, image of vocational 
education): 


Positive results overall or for particular program 
areas would have implications for school counsel* 
ing and vocational program improvement, It 
could also help define the basic mission of voca- 
tional education. Public support could increase. 


(13) 


Dissemination of findings 
and suggested applications: 


Findings should be shared with SEAs, NACVE, 
SACVEs, and RCU network and disseminated 
through other appropriate channels. 


(14) 


Implications of findings for 
further RD&E activities and 
relationship of this evalu- 
ation to previous studies: 


Similar studies should be done in a number of 
SEAs and LEAs, perhaps using different instru- 
ments. Procedures and findings should be com- 
pared with previous studies, including those 
based on data from the high school class of 1972,^ 


Notes: * See abbreviations at end of table. 
^ See notes at end of table. 
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TaWe4. SOME KEY OUTCOMEji FOR USE IN EVALUATING 



OUTCOME NO. 2 • DEVELOPMENT OF USEFUL OCCUPATIONAL SKILLS 


ESSENTIAL ELEMENTS OF INFORMATION FOR AN EVALUATION STUDY 




Checklist of Elements 


Information for This Outcome* 


(1) 


Outcome question to be 
answered: 


Do Studsnts acnuirB ucpful nrrunatinnal cl^lllc 

by participating in school-based vocational 
programs? 


(2) 


Outconne hypothesized as 
the dependent variable: 


Development of useful occupational skills. 


(3) 


Affected entity (student 
population or other entity 
in which the outcome 
is observed): 


Studentn who have completed at least 50 percent 
of an approved high school vocational program 
during a specified time period. 


(4) 


Prograrn identified as the 
strategic independent variable: 


Completion of at least 50 percent of an approved 
high school vocational program (i.e., sequence of 
courses and related experiences approved by the 
state vocational education agency). 


(5) 


Rationale for hypothesized 
outcome and appropriateness 
of the outcome as a basis 
for evaluating the program: 


Vocational programs provide instruction in a 
variety of occupational skills presumed to be 
useful for today's iobs. esoeclallv in entrv-lpwpl 
employment. While acknowledging other poten- 
tial benefits of vocational education, many people 
regard the development of occupational skills as 
the central focus of the curriculum and the 
outcome over which the school has the greatest 
control. 


(6) 


Empirical indicators 
of the outcome: 


Cognitive competencies and performance of a 
valid sample of tasks that entry-level workers are 
required to do in particular occupation.:!. 


(7) 


Methodological considerations 
(evaluation design, sampling 
procedures, statistical analysis): 


Before-and-after design. Random samples can be 
drawn from high school students enrolled in one 
or more (a) occupational areas, (b) schools, (c) 
local school systems, (d) states. Statistical tests 
of the significance of gains. 


(8) 


Evaluation standard (basis for 
interpreting outcome data to 

judge program effectiveness): 


Attainment of significant gain scores and post 
scores that achieve minimum performance 
required for entry-level workers. 









24 



VOCATIONAL EDUCATION PROGRAMS Outcome No. 2 





Checklist of Elements 


Information for This Outcome* 


(9) 


Data requirements (instru- 
ments, procedureis, data base): 


Scores on standardized occupational skill per- 
formance tests or proxies for them' (e.g., written 
tests intended to measure skills indirectly) and 
judgments by vocational Instructors and employer 
panels on scores required for minimum entry-level 
performance. 


(10) 


Feasibility of investigating 
this outcome (conceptual, 
administrative, cost, time, 
other considerations): 


In the absence of valid standardized performance 
tests, data will be highly judgmental (subject to 
bias and other types of error), thereby under- 
mining credibility of the findings. Administrative 
burdens could be substantial. Lack of conceptual 
basis for comparing vocational vs. nonvocational 
students in terms of the outcome variable makes 
it difficult to identify an appropriate evaluation 
standard. 


(11) 


Findings (results of data 
analysis, warranted infer- 
ences, generalizability): 


Do the outcome data indicate that students 
acquire useful occupational skills by participating 
in. high school vocational programs and, if so, to 
what extent? Can the findings be generalized to 
other populations? 


(12) 


Potential impact of findings 
(implications for policy, 
program design and manage- 
ment, image of vocational 
education): 


Credible evidence that youth acquire useful 
occupational skills through high school voca- 
tional programs will demonstrate a tangible out- 
come, the value of which depends on labor 
market conditions and other factors largely 
beyond control of the schools. 


(13) 


Dissemination of findings 
and suggested applications: 


Findmgs should be shared with SEAs, NACVE, 
SACVEs, and RCU network and disseminated 
through other appropriate channels. 


(14) 


Implications of findings for 
further RD&E activities and 
relationship of this evalu- 
atior> to previous studies: 


The study should be replicated with the same 
and also other (a) occupational programs and 
(b) SEAs and LEAs. Emphasis should be placed 
on developing effective data-collection instru- 
ments and procedures." 


Notes: * See abbreviations at end of table. 
' See notes at end of table. 
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SOME KEY OUTCOMES FOR USE IN EVALUATING 



OUTCOME NO. 3 • REDUCING THE RISK OF UNEMPLOYMENT 

FOR MINORITY YOUTH 



ESSENTIAL ELEMENTS OF INFORMATION FOR AN EVALUATION STUDY 





Checklist of Elements 


Information for This Outcome* 


(1) 


Outcome question to be 
answered: 


Does completion of a high school vocational 
program reduce the risk of unemployment for 
minority youth? 


U) 


Outcome hypothesized as 
the dependent variable: 


Reduction in the risk of unemployment. 


(3) 


Affected entity (student 
population or other entity 
in which the outcome 
IS observed): 


Minority youth who completed a high school 
vocational program and were labor force 
participants half a year later. 


/ (4) 


Program identified as the 
strategic independent variable: 


Completion of an approved vocational program 
at the secondary level during a specified school 
year. 


(5) 


Rationale for hypothesized 
outcome and appropriateness 
of the outcome as a basis for 
evaluating the program: 


Since vocational programs provide instruction, in 
occupational skills and afford other employment- 
related experiences, it is presumed that students 
will acquire capabilities that make thern more 
attractive to employers, Job-readiness is a desired 
characteristic of vocational program completers. 


(6) 


Empirical indicators of the 
outcome: 


Unemployment rate calculated for the program 
population (i.e., number unemployed as a per- 
centage of the active labor force). This indicates 
the statistical probability of an ''average" worker 
in the program population being unemployed, 


(7) 


Methodological considerations 
(evaluation design, sampling 
procedures, statistical analysis): 


Comparison group design. Sampling procedure as 
used in SVEA information system, Calculation 
of confidence intervals for unemployment rates. 




LVuiuaiion stanuaro (c>asis tor 
interpreting outcome data to 
judge program effectiveness): 


Lower calculated unemployment rate for the 
program population than for one or more com- 
parison groups. 


(9) 


Data requirements (instru- 
ments, procedures, data base): 


Labor force status of the program population and 
comparison group (or proxy) approximately six 
to eight months following the end of the school 
year. 
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VOCATIONAL EDUCATION PROGRAMS Outcome No. 3 





Checklist of Elements 


Information for T^^'«: Outcome* 


(10) 


Feasibility of investigating 
this outcome (conceptual, 
administrative, cost, time, 
other considerations): 


Unemployment rate of program population can 
be calculated from existing data, although the 
accuracy of reporting may be uncertain. Unless 
special survey data are available or can be 
collected, the comparative evaluation standard 
will be of limited validity. The relative importance 
of the outcome itself may be questioned in some 
SEAs and LEAs. 


(11) 


Findings (results of data 
analysis, warranted infer- 
ences, gsneralizability): 


Do unemployment rates of the program popula- 
tion and comparison group indicate differences 
that can be attributed to program completion? 
Can the findings be generalized to other popula- 
tions? 


(12) 


Potential impact of findings 
(implications for policy, 
program design and, manage- 
ment, image of vocational 
education): 


Lower unemployment rates for vocational com- 
pleters would suggest school-based vocational 
education may be an effective policy to reduce 
minority youth unemployment and should be 
extended to more students. Finding no difference 
or higher rates suggests a need to determine why. 


(13) 


Dissemination of findings 
and suggested applications: 


Findings should be share J with SEAs, NACVE, 
SACVEs, and RCU network and disseminated 
through other appropriate channels. 


(14) 


Implications of findings for 
further RD&E activities and 
relationship of this evalu- 
ation to previous studies: 


The study should be replicated in other SEAs 
and LEAs, with other populations and in other 
time periods. Procedures and findings should be 
compared with past studies." 


Notes: * See abbreviations at end of table. 
^ See notes at end of table. 
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Table 4. SOME KEY OUTCOMES FOR USE IN EVALUATING 



OUTCOME NO. 4 • ACQUISITION OF WORLD-OF-WORK KNOWLEDGE 


ESSENTIAL ELEMENTS OF INFORMATION FOR AN EVALUATION STUDY 




Checklist of Elements 


Information for This Outcome* 


(1) 


. Outcome question to be 
answered: 


Do vocational students acquire greater world-of- 
wu^^ Knowieuge man comparaoie nonvocationai 
students? 


(2) 


Outcome hypothesized as 
the dependent variable: 


Acquisition of world^of-work knowledge. 


(3) 


Affected pntitx/ (^tuHpnt 
population or other entity 
in which the outcome 
is observed): 


Students who have completed a minimum of 
one and one-half semesters of high school or 
postsecondary occupational education. 


14) 


Program identified as the 
strategic independent variable: 


Completion of at least one and one-half semesters ' 
of a vocational program at the high school level 
or in a postsecondary vocational-technical insti- 
tute, community college, or other postsecondary, 
nonbaccalaureate institution. 


(5) 


Rationale for hypothesised 
outcome and appropriateness 
of the outcome as a basis for 
evaluating the program: 


In its efforts to enhance the employability, 
productivity, and earnings of students, vocational 
education is typically concerned with a wider 
range of employment-related topics than occupa- 
tional skills alone. One of these can be termed 
woria-oT-worK Knowieuge, wnicn includes! 
occupational information, career planning, knowl- 
edge of labor market processes, job-search 
strategies, and work-adjustment skills. There is 
some research evidence indicating that labor 
market outcomes are influenced by factors such 
as inese. vocational programs may be expected 
to generate higher levels of world-of-work knowl- 
edge than academic programs, which focus on 
other educational objectives. 


(6) 


Empirical indicators of 
the outcome: 


Scores on cognitive tests of occupational informa- 
tion ioh-sparch mpthnris sknd l;)hnr mar^Pt 

processes; and performance of tasks in the areas 
of decision making, job interviewing, career plan- 
ning, and work adjustment. 


(7) 


Methodological considerations 
(evaluation design, sampling 
procoduros, statistical analysis): 


Comparison-group design. Stratified random 
samples of program population and comparison 
group. Statistical tests of the significance of 
differences. 
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VOCATIONAL EDUCATION PROGRAMS 



Outcome No. 4 





Checklist of Elements 


Information for This Outcome* 


(8) 


Evaluation standard (basis for 
interpreting outcorne data to 
judge program effectiveness) : 


Superior performance of vocational students vs. 
nonvocational students matched as nearly as 
possible for race/ethnicity, mental ability, socio* 
economic background, sex, quantity of schooling, 
and work experience, 


(9) 


Data requirements (instru- 
ments, procedures, data base): 


Scores on cognitive and performance tests as well 
as background data will be required on students 
in program and comparison groups. Existing 
instruments such as the NLS occupational infor- 
mation test can perhaps be used. Procedures for 
assessing performance can^be developed (drawing 
on CETA models). 


(10) 


Feasibilitv of investiaatina 
this outcome (conceptual, 
administrative, cost, time, 
other considerations): 


This outcom6 ranks lower in feasibilitv than snmp 
Others because (a) the meaning attached to the 
outcome is neither completely clear nor accented, 

(b) valid instruments are not readily available, and 

(c) it may be difficult to generate data for a satis- 
factorily matched comparison group, 


(11) 


Findings (results of data 
analysis, warranted infer- 
ences, general izability): 


Positive findings can be interpreted as evidence 
that vocational progiams generate employment- 
related benefits beyond the development of 
occupational skills, which are sometimes alleged 
to be narrow and obsolescent-prone. 


(12) 


Potential impact of findings 
(implications for policy, 
program design and manage- 
ment, image of vocational 
education): 


Such ? finding might encourage high schools to 
strengthen the world-of-work-information com- 
ponent of their nonvocational programs. If voca- 
tional students do not compare favorably in 
world-of-work knowledge, the objectives and 
processes should be reviewed to determine whit 
program changes, if any, should be made. 


:i3) 


Dissemination of findings 
and suggested applications: 


Findings should be shared with SEAs, NACVE, 
SACVEs, 8nd RCU network and disseminated 
through other appropriate channels. 


(14) 


Implications of fin.^ings for 
further RD&E activities and 
relationship of this evalu- 
ation to previous studies: 


The study should be replicated. Instruments and 
procedures should be critiqued and improved 
reflecting increased understanding of the out- 
come. Comparison could be made with the NLS 
findings and other studies.^ 


Notes; * See abbreviations at end of table. 
" See notes at end of table. 
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SOME KEY OUTCOMES FOR USE IN EVALUATING 



OUTCOME NO. 5 • EDUCATIONAL COMMITMENT AT THE HIGH SCHOOL LEVEL 


1 


ESSENTIAL ELEMENTS OF INFORMATION FOR AN EVALUATION STUDY 




Checklist of Elements 


Information for This Outcome* 


(1) 

> 


Outcome question to be 
answered: 


Does enrollment in a vocational program 
strengthen the educational commitment of 
economically disadvantaged students at the high 
school level? 


(2) 


Outcome hypothesized as 
the dependent variable: 


Educational commitment of economically 
disadvantaged students. 


(3) 


Affected entity (student 
population or other entity 
in which the outcome 
is observed): 


Economically disadvantaged high school students 
who enroll in a vocational program. 


14) 


Program identified as the 
strategic independent variable: 


Enrollment in an approved vocational program at 
the 10th, 11th, or '12th grade level and attend- 
ance of at least 60 percent of vocational classes 

riurinn tlio firct four lAiool/e 


(5) 


Rationale for hypothesized 
outcome and appropriateness 
of the outcome as a basis for 
evaluating the program: 


By teaching entry-level cx:cupational skills, voca- 
tional programs offer students the prospect of 
employment, earnings, and economic self-reliance 
in the near future. The immediacy of this return^ 


(6) 


Empirical indicators of 
the outcome* 


School attendance, grades, quantity of schooling 
compieiea, graouaiion rates, ana attitudinal 
indicators. 


(7) 


Methodological considerations 
(evaluation design, sampling 

nrnPPfllirPQ ct;itlctipal ^r')aiv/cic\ * 
^'1 WWCUUI C9.> dLullbllLdI dfioiyalc/. 


Comparison-group and before-and-after design. 
Stratified random samples of economically dis- 
aavaniagea students oy enrollment in vocational 
education or general/college preparatory curric- 
ula. Statistical tests of differences between the 
two groups and within the vocational group, 
before and after enrollment. 


(8) 


Evaluation standard (basis for 
interpreting outcome data to 
judge program effectiveness): 


Superior attendance, arades auantitv nf ^rhnni- 
ing completed, graduation rates, and attitudes 
towards school, work, and life of vocational 
students vs. nonvocational students. 


(9) 

1 


Data requirements (instru- 
n)ents, procedures, data base): 


Data required on student background, participa- 
tion in vocational education, and outcome 
indicators listed in (6) above. 
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VOCATIONAL EDUCATION PROGRAMS Outcome No. 5 





Checklist of Elements 


Information for This Outcome* 


(10) 


Feasibility of investigating 
this outcome (conceptual, 
administrative, cost, time, 
other considerations): 


This evaluation study would require substantia! 
staff resources, time (longKudinal data over a 
period of two or three years), and strong admin- 
istrative support. Without these elements, its 
feasibility is questionable. / 


(11) 


Findings (results of data 
analysis, warranted infer- 
ences, generalizability): 


Finding that enrollment in vocational education 
strengthens the educational commitment of high 
school students would document an outcome of 
broader significance than job placement and 
other immediate labor market benefits. Gradua- 
tion from high school is socially and economically 
valuable per se and also is typically a requirement 
for postsecondary education, including pursuit of 
a baccalaureate degree. 


(12) 


Potential impact of findings 
(implications for policy, 
program design and manage- 
ment, image of vocational 
education): 


Positive findings would strengthen the reputation 
of vocational education as an effective alternative 
to the academic and general high school curricula. 
Differential outcomes by vocational program area 
could suggest ways of improving various programs. 


(13) 


Dissemination of findings 
and suggested applications: 


Procedures and findings could be share?! with 
evaluators in other LcAs and SEAs, and results 
could bo reported in professional journals and 
public media. 


(14) 


Implications of findings for 
further RD&E activities and 
relationship of this evalu- 
ation to previous studies: 


The study should be replicated in other school 
systems. Differential outcomes by program area, 
sex, age, and race/ethnicity should be studied in 
greater depth. ^ 


Notes: * See abbreviations at end of table. 
^ Continued at end of table. 
^ See notes at end of table. 
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SOME KEY OUTCOMES FOR USE IN EVALUATING 



OUTCOME NO 6 


• DEVELOPMENT OF LEADERSHIP QUALITIES 


ESSENTIAL ELEMENTS OF INFORMATION FOR AN EVALUATION STUDY 




Checklist of Elements 


Information for This Outcome* 


(1) 


Outcome question to be 
answered: 


Does enrollment in a high school vocational 
program promote the development of leadership 
qualities? 


(2) 


Outcome hypothesized as 
the dependent variable: 


Development of leadership qualities. 


(3) 


Affected entity (student 
population or other entity 
in which the outcome 
is observed): 


Students who for one school year or more, at the 
secondary level, (a) wfire enrolled in an approved 
vocational program and (b) actively participated 
in a vocational student organization (VSO). 


(4) 


Program identified as the 
strategic independent variable: 


The combination of enrollment in a high school 
vocational program and participation in a VSO 
as specified in (3} above. 


(5) 


Rationale for hypothesized 
outcome and appropriateness 
of the outcome as a basis for 
evaluating the program: 


Many vocational students are encouraged to join 
and become actively involved in VSOs. These 
groups prc 'ide opportunities for social inter- 
action, occupationally related activities, and 
. organizational experience for students outside 
the classroom. There is an atmosphere of friend- 
liness and security fostered by homogeneity of 
occupational interests. The need for student 
leaders to carry out activities of the organizations 
creates practical opportunities for leadership 
development. VSOs may afford special leader- 
ship opportunities for minority and female 
students. 


(6) 


Empirical indicators of 
the outcome: 


Activities and achievements identified by experts 
as evidence of leadership qualities. 


(7) 


Methodological considerations 
(evaluation design, sampling 

nrorodurPQ Qtati<:tir;il annlwQicI' 


Retrospective case study design. Purpos've 
sample designed to overrepresent high achievers; 
^irdiiritfu uy occupoiiunai iieius, race/etnnicity, 
sex, and age (e.g., 16-19 years, 20-24, 25-35, 
over 35). Major reliance will be placed. on non- 
statistical analyses of differences. 


(8) 


Evaluation standard (basis for 
interpreting outcome data to 
judge program effectiveness): 


Interpretation of overall results will rely on a 
judgment about the leadership qualities attained 
due to participation in high school vocational 
programs. Comparisoni^n be made among the 
various subgroups. 
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VOCATIONAL EDUCATION PROGRAMS 



Outcome No. 6 





Checklist of Elements 


Information for This Outcome* 


(9) 


Data requirements (instru- 
ments, procedures, data base): 


Background information and data on leadership 
experiences and achievements of the program 
students will be required. 


(10) 


Feasibility of investigating this 
outcome (conceptual, admin- 
istrative, cost, time, other 
considsrations): 


Use of the case study method ririakes this out- 
come administratively feasible, although it could 
be quite costly if the scope of the study is broad. 
Credibility could b. j problem, tn the absence 
of an experimental or comparison-group design, 
positive findings may not be convincing to some 
audiences. 


(11) 


Findings (results of data 
analysis, warranted infer- 
ences, generaiizaDiiiiy;. 


Within the vocational education community it is 
assumed on faith that leadership skills and self- 
image are notably enhanced through participation 
in student organizations. Docunr.entation of this 
outcome would pot only have value per se but 
also would illustrate the evaluation of vocational 
programs on the basis of an outcome that is quite 
different from job placement. j 


(12) 


Potential impact of findings 
(implications for policy, 
program design and manage- 
ment, image of vocational 
education): 


Positive findings could demonstrate that voca- 
tional education generates outcomes beyond 
individual labor'market benefits. Such findings 
might influence both policy and program planning. 


(13) 


Dissefnination of findings 
and suggested applications: 


Success stories, including cases involving women 
and minorities, could be used to generate feature 
articles for newspapers, magazines, and other 
media. Findings could be disseminated through 
advisory councils and professional associations to 
expand awareness of vocational education as a 
mechanism for promoting upward mobility. 


(14) 


Implications of findings for 
further RD&E activities and 
relationship of this evalu- 
ation to past studies: 


The studies could be replicated by SEAs and LEAs 
with emphasis on ' r own personnel. Methods 
might be de\/elopeo :o provide a basis for com- 
paring the leadership, effects of vocational pro- 
grams with those of another program experience, 
rariicuiar components couio De examinee to 
determine how they did or did not contribute to 
affective, cognitive, and behavioral aspects of 
leadership. The case study methodology could 
be applied to other outcomes that are not 
conducive to statistical design.^ 


Nolos 




* See abbrevi'itions at end of table. 
^ See notes at end of table. 
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SOME KEY OUTCOMES FOR USE IN EVALUATING 



OUTCOME NO. 7 


• POSTSECONDARY EDUCATIONAL SUCCESS 


E 


SSENTIAL ELEMENTS OF INFORMATION FOR AN EVALUATION STUDY 




Checklist of Elements 


Information for This Outcome* 


(1) 


Outcome question to be ^ 
answered: 


What effect does completion of a high school 
^ vocational program have on access tc and 
success in various forms of postsecondary 
education and training? 


(2) 


Outcome hypothesized as 
the dependent variable: 


Success in postsecondary education and training. 




Affected entity (student 
population or other entity 
in whicff the outcome 
is observed): 


Recent high school graduates who completed a 
high school vocational program and either (a) 
were admitted to or (b) unsuccessfully sought 
admission to postsecondary education or training. 


(4) 


Program identified as the 
strategic independent variable: 


Completion of an approved high school vocational 
program and graduation from high school within 
the past six years. 


(5) 


Rationale for hypothesized 
outcome and appropriateness 
of the outcome as a basis for 
evaluating the program: 


Unlike the college preparatory curriculum, the 
major purpose of vocational education is not to 
prepare students for entry into postsecondary 
ed- -national programs. Many vocational students, 
h. .„ever, expect to pursue their schooling or 
training beyond high school and in fact do so, 
whether in a community college, technical insti- 
tute, four-year college or university, or appren- 
ticeship program.^ 


(6) 


Empirical indicators of 
the outcome: 


Admission rates, grades, and completion rates. 


(7) 


Methodological considerations 
(evaluation design, sampling 
procedures, statistical analysis): 


Comparison-group design. Stratified random 
samples of vocational completers/graduates am. 
graduates of other high school curricula. Sta' 
tical tests of the significance of differences. 


(8) 


Evaluation standard (basis for 
interpreting outcome data to 
judge program effectiveness) : 


'-kuiiiiaaiun rales, graues, ana completion rates tor 
a sample of recent high school vocational gradu- 
ates (controlled for age, sex, race/ethnicity, 
socioeconomic factors) that are (a) not lower than 
for nonvocational graduates with respect to bacca- 
laureate programs and (b) higher than nonvoca- 
tional graduates in nonbaccalaureate programs. 
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VOCATIONAL EDUCATION PROGRAMS Outcome No. 7 





Checklist of Elements 


• ^ ^ 

Information for This Outcome* — ^ 


(9) 


Data requirements (instru- 
ments, procedures, data base): 


• Data from student interviews and school riecords 
as specified in (8) above. 


(10) 


Feasibility of investigating 
this outcome (conceptual,;}, 
administrative, cost, t^rhe, 
other considerations): 


This study is administratively feasible but will 
require a minimum of one year to collect longi- 
tudinal data and will be costly (because of 
student interviews and examination of school 
records). 


(11) 


Findings (results of data 
analysis, warranted infer- 

c^nppQ nPDPrj) 1 i7;)Hi 1 i t\/ ) * 
CI yci ici ai i^dui 1 1 Ly / . 


If it is found that vocational education contributes 
to studrnt suc-.ess in postsecondary occupational 
cuuLaiion ana aoes noi inniuii success in lour^ycar 
postsecondary programs, this will be seen as an 
important outcome. Such a finding would refute 
the assumption that high school vocational educa- 
tion tracks students irreversibly into subprofes- 
sional careers by limiting their access to college. 


(12) 


Potential impact of findings 
(implications for policy, 
program design and manage- 
ment, imshae of vocational 
GoucoLion / .\ 


Findings as described in (11) could tend to 
increase vocational enrollments at the high school 
level, attracting more students^tro may be plan- 
ning to enter a four-year college. This could under- 
mme ine traoiiionai mission ot vocational eouca- 
tion, to prepare students for entry-level job place- 
ment in a field related to their training. Evidence 
that refutes the "tracking" criticism could tKere- 
Tore inauce cnanges in admission stanoaras, 
program goals, and accountability criteria. 


(13) 


Dissemination of findings 
and suggested applications: 


Findings should be shared with the RCU network 
and disseminated through other appropriate 
channels. 


(14) 


Implications of findings for 
further RD&E activities and 
relationship of this evalu- 
ation to prpvious studies: 


Similar studies should be done in other SEAs and 
LEAs. Data-collection methods should be varied. 
Studies could focus on differential effects of 
particular vocational program areas and on longer 
time spans.^ 


Notes: * See abbreviations at end of table. 
Continued at end of table. 
^ See notes at end of table. 
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SOME KEY OUTCOMES FOR USE IN EVALUATING 



HYPOTHESIZED OUTCOME NO. 8 


• HIGHER EARNINGS 


E 


SSENTIAL ELEMENTS OF INFORMATION FOR AN EVALUATION STUDY 




Checklist of Elements 


Information for This Outcome* 


(1) 


Outcome question to be 
answered: 


What effect does vocational education have on 
the earnings of young workers during their first 
two years after leaving school? 


(2) 


Outcome hypothesized as /' 
the dependent variable: - 


Higher earnings than would have been forthcoming 
if the student had not enrolled in a vocational 
program. 


(3) 


Affected entity (student 
population or other/fentity 
in which the outcome 
is observed): 


Students who completed 50 percent or more of 
an approved vocational program (high school or 
postsecondary) during a specified time period, 
whether they subsequently graduated or dropped 
out. 


(4) 


Program identified as the 
strategic independent variable: 


Completion of 50 percent or more of an approved 
vocational program (i.e., sequence of courses and 
related experiences approved by the SVEA). 


(5) 


Rationale for hypothesized 
outcome and appropriateness 
of the outcome as a basis for 
evaluating the program: 


Vocational programs purport to teach job skills 
and prepare students for productive employment. 
In a market economy, workers with superior 
skills are assumed to be more productive and to 
earn more than those who are less skilled. Occupa- 
tional training should give vocational students an 
early labor market advantage over comparable 
students who have not had job-related training. 


(6) 


E:mpirical indicators of 
the outcome: 


Hourly wages as of particular time periods (e.g., 
eight months after leaving school and again 
sixteen months later). 


\ // 


Methodological considerations 
(evaluation design, sampling 
procedures, statistical analysis); 


Comparison-group design. Stratified random 
samples can be drawn from program population 
in different occupational areas, schools, and labor 
markets, excluding persons who received more 
tnan bu nours or postscnooi traming to qualify 
for job assignment. Statistical tests will be per- 
formed to determine significance oi differences 
in hourly wages. 




Evaluation standard (basis for 
intorpreting outcome data to 
judge program effectiveness): 


Higher mean hourly wages of the program popu- 
lation than for a group of workers having charac- 
teristics similar in all respects except that they did 
not paiticipate in « ocational education. 
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VOCATIONAL EDUCATION PROGRAMS Hypothesized Outcome No. 8 





Checklist of Elements* 


Information for This Outcome* 


(9) 


Data requirennents (instru- 
nnents, procedures, data base): 


Background information on program population 
and comparison group plus hourly earnings data 
for a given time period and given labor market 
areas. One possible source of wage data is the 
self-reporting follow-up survey conducted by 
LEAs under VEDS. These data could be verified 
with employers for c sample of respondents. 


(10) 


Feasibility of investigating 
this outconne (conceptual, 
administrative, cost, tinne, 
other considerations): 


All aspects of this investigation appear highly 
feasible with the possible exception of compari- 
son group data. An alternative evaluation standard 
might have to be used based on general labor 
force statistics. 


(11) 


Findings (results of data 
analysis, warranted infer- 
ences, general izability): 


Observation and statistical analysis of the desig- 
nated empirical indicator will provide a partial 
answer to the outcome question. Generalization 
beyond the labor market area studied would not 
be warranted. 


(12) 


Potential impact of findings 
(implications for policy, 
program design and mariage-. 
ment, image of vocatfonal 
education): 


Positive findings would contradict many national 
studies, which report no earnings advantage for 
former vocational students. They would suggest 
that the local program "pays off and may be 
superior to vocational programs in some other 
states. 


(13) 


Dissemination of findings 
and suggested applications: 


Findings should be shared with SEAs, NACVE, 
SACVEs, and RCU network and disseminated 
through other appropriate channels. 


(14) • 


Implications of findings for 
further RD&E activities and 
relationship of this evalu 
ation to previous studies: 


Since this outcome will be studied in all states as 
part of VEDS, good opportunities will exist for 
comparing data-collection procedures, responses, 
and interpretations of results. The methodology 
and findings should also be compared with previ- 
ous national studies to determine why the results 
differed.^ 


Notes: * See abbreviations at end of table. 
^ See notes at end of table. 
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Table 4. 



SOME KEY OUTCOMES FOR USE IN EVALUATING 



1 OUTCOME NO. g 


• STUDENT SATISFACTION WITH SCHOOL 


1 ^ 


SSENTIAL ELEMENTS OF INFORMATION FOR AN EVALUATION STUDY 




J Checklist of Elements 


Information for This Outcome* 


(1) 


Outcome question to be 
answered; 


Do students enrolled in vocational programs 
express more, less, or the same satisfaction with 
their current school experience as comparable 
nonvocational students? 


1 (2) 


j uuTcome nypotnesized as 
the dependent variable: 


Student satisfaction with current school experi- 
ence, whether high school or postsecondary. 


'3) 


Affected entity (student 
population or other entity 
in which the outcome 
is observed): 


Students currently enrolled in an occupational 
curriculum, whether at the secondary or post- 
secondary level. 


(4) 


Program identified as the 
strategic independent variable: 


Enrollment in c school-based vocational program, 
whether secondary or postsecondary. 


(5) 


Rationale for hypothesized 
outcome and appropriateness 
of the outcome as a basis for 
evaluating the program: 


Outcomes selected as criteria for evaluating voca- 
tional programs typically have an investment 
orientation rather than a focus on direct satis- 
faction to students.^ Student satisfaction with 
school is an outcome that has direct consumer 
value, particularly to the individual student, as " 
well as Indirect investment-type benefits through 
its motivational effects on quality and quantity 
of school achievement. U.S. society values the 
pursuit of happiness and quality of life. Since 
schooling occupies a central place In the lives of 
young people, the enjoyment they derive from 
school is an appropriate criterion, among others, 
for evaluating the school program. 


(6) 


Empirical indicators of 
the outcome: 


Responses to attitudinal questions that directly 
and indirectly measure satisfaction with various 
facets of the current school experience. 


(7) 1 


Methodological considerations 
(evaluation design, sampling 
procedures, statistical analysis): 


samples of secondary and postsecondary students 
in a given state or local community. 


(8) 

1 


Evaluation standard (basis for 
interpreting outcome data to 
judge program effectiveness); 

J 


More positive attitudes held by vocational 
students than by nonvocational students, at 
secondary and postsecondary levels, matched 
for sex, age, mental ability, and socioeconomic 
background. 
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VOCATIONAL EDUCATION PROGRAMS Outcome No. 9 





Checklist of Elennents 


Information for This Outcome* 


(9) 


Data requirements (instru- 
ments, procedures, data b<ise): 


Background data and questionnaire responses 
required for vocational students and comparison 
groups. Possible use of NLS instruments. 


(10) 


Feasibility of investigating 
this outcome (conceptual, 
administrative, cost, time, 
other considerations): 


Using a relatively naive design, this study is highly 
feasible both administratively and in terms of 
. cost If a methodologically sophisticated study is 

1 1 nnprtr)l< Pn pHfnInlQtrsttlwo nroKlomc onrJ /^octc 
uriucf larvcri, auiiiiriisiiaiivc fJiUUIcrnb anu LOSlS 

will escalate. In either case, questions will arise 
with respect to the validity of questionnaire items, 
matching of vocational vs, nonvocational students, 
and the basic rationale for valuing student satis- 
faction. 


(11) 


Findings (results of data 
analysis, warranted infer- 
ences, generalizability): 


There is a great deal of uncertainty and disagree- 
ment among educators, legislators, and the general 
public regarding the purposes and desirable out- 
Vi«uriica \j\ cuuuaiion, M TinQing inai vocaiionai stu- 
dents enjoy what they are studying more than they 
would enjoy alternative educational programs 
(regardless of whether vocational programs gener- 
ate labor market benefits) would generally be 
regarded as a positive outcome. 


(12) 


Potential impact of findings 
(implications for policy, 
program design and manage- 
ment, image of vocational 
pduration) ■ 


Positive findings could increase student interest in 
vocational education and public support for 
expansion of the program. 


(13) 


Dissemination of findings 
and suggested applications: 


Share instruments, procedures, and results with 
other SEAs and LEAs. Other dissemination 
dependent on credibility of the study and nature 
of findings. 


(14) 


Implications of findings for 
further RD&E activities and 
relationship of this evalu- 
ation to previous studies: 


Compare findings and methodology with previous 
studies and explain important differences. Repli- 
cate in other states and local communities. 
Examine in depth the programmatic and other 
correlates of high satisfaction,^' 


Notes: * See abbreviations at end of table. 
''•^ See note.s at end of table. 
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SOME KEY OUTCOMES FOR USE IN EVALUATING 



OUTCOME NO. 10 


• REDUCTION OF JOB-SEARCH TIME 


E 


SSENTIAL ELEMENTS OF INFORMATION FOR AN EVALUATION STUDY 




Checklist of Elements 


Information for This Outcome* 


(1) 


Outcome question to be 
answered: 


What effect does vocational education have on the 
length of time it takes young workers to find full- 
lime employment after leaving high school? 


(2) 


Outcome hypothesized as 
the deoendent 


Reduction of job-search time. 


(3) 


Affected entity (student 
population or other entity 
in which the outcome 
is observed): 


Recent high school leavers (whether graduates or 
dropouts) who completed a minimum of one full 
semester of an approved vocational program and 
entered the labor force seeking full-time employ- 
ment immediately after leaving school. 


(4) 


Program identified as the 
strategic independent variable: 


Completion of one semester or more of an 
approved high school vocational program. 


(5) 


Rationale for hypothesized 
outcome and appropriateness 
of the outcome as a basis for 
uvcjiuaiing ine program. 


Development of occupational skills and exposure 
to world-of-work information and experiences as 
a participant in vocational education can be 
expected to enhance the job-seeking ability of 
young workers. The length of time it takes to find 
a job is one test of this ability. Vocational pro- 
grams often provide direct assistance in job place- 
ment. Assuming compensation at the federal 
minimum wage of $3.10 per hour, the economic 
value of reducing job-search time for vocational 
students by as little as two weeks would be $248. 
This amount represents a significant benefit of 
vocational education. 


(6) 


Empirical indicators of 
the outcome: 


Number of days (weeks) spent searching for 
employment after leaving school, i.e., from date 
of leaving school to date of securing regular, 
full-time employment, not first day at work on 
the job. (Students hired for a regular, full-time 

1 ^^^^ to ro lost/ 1 n n cf> \r\ r\fy\ t ai ■ 1 1 V> •^^••^ * ^in^ 

juu utfiuic it^dviny scnool will nave zero jou-searcn 
time as here defined.) 


(7) 


Methodological considerations 
(evaluation design, sampling 
procedures, statistical analysis): 


Comparison-group design. Stratified random 
samples of former vocational students and com- 
parable former nonvocational students. Statistical 
tests of the significance of differences. 
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VOCATIONAL EDUCATION PROGRAMS Outcome No. 10 





T-^ 

Checklist of Elements 


Information for This Outcome* 


(8) 


Evaluation standard (basis for 
interpreting outcome data to 
judge program effectiveness): 


Less time spent by vocational students searching 
for a job as here defined than by nonvocational 
students. 


(9) 


Data requirements (instru- 
ments, procedures, data base): 


Background data from school records and job- 
search time and supplementary information 
reported by students in the sample. Use might be 
made of data from the new youth cohort of the 
National Longitudinal Surveys (NLS), but supple- 
mentary interview questions would be required 
to distinguish clearly between time spent in volun- 
tary idleness versus involuntary unemployment.^ 


(10) 


Feasibility of investigating 
this outcome (conceptual, 
administrative, cost, time, 
other considerations): 


Unless existing data are used, such as the new 
' NLS youth cohort (which itself may not be fully 
adequate for purposes of this particular type of 
search-time study), the costs of this study would 
be substantial. 


(11) 


Findings (results of data 
analysis, warranted infer- 
ences, generalizability): 


A finding that vocational education reduces job- 
search time will be seen as a documented labor 
market benefit. 


(12) 


Potential impact of findings 
(implications for policy, 
program design and manage- 
ment, image of vocational 
education): 


Positive findings would reinforce programs that 
contribute to the enhancement of job-search 
skills. Nonvocational programs might emulate 
successful vocational practices. 


(13) 


Dissemination of findings and 
suggested applications: 


Findings should be shared with RCU network and 
disseminated through other appropriate channels. 


(14) 


Implications of findings for 
further RD&E activities and 
relationship of this evalu- 
ation to previous studies: 


The study should be replicated. Instruments and 
procedures should be critiqued and improved, 
reflecting increased understanding of the out- 
come. Comparison could be made with the NLS 
findings and other studies.^ 


Notes: * See abbreviations at end of table. 
^•^ See notes at end of table. 
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Table 4. SOME KEY OUTCOMES FOR USE IN EVALUATING 



OUTCOME NO. 11 


SATISFACTORINESS TO EMPLOYERS 


\ 

ESSENTIAL ELEMENTS OF INFORMATION'^OR AN EVALUATION STUDY 




Checklist of Elements 


\|iformation for This Outcome* 


(1) 


Outcome question to be 

dl laWcrcU • 


To what exterV^ do employers consider former 
vocational siuaenis to oe wen trainee ano 
prepared for employment? 




the dependent variable: 


DatisTocxoriness oi vocational programs as 
perceived by employers. 
. \ 


(3) 


Affected entity (student 
population or other entity 
in which the outcome 

la OUaci VcQ / . 


Employers of former vbcational students (as 
influenced by performance of employees who 
took vocational education). 


(4) 


Program identified as the 
strategic independent variable: 


Secondary and postsecondary vocational programs 
that purport to impart entn/-level job skills. 


(5) 


Rationale for hypothesized 
outcome and appropriateness 
of the outcome as a bas«s for 
evaluating the program: 


To the extent that vocational programs purport- 
ing to teach entry-level job skills are successful, 
they will turn out program completers and leavers 
who are likely to be considered by their employers 
to be well trained and prepared for employment. 


(6) 


Empirical indicators of 
the outcome: 


Employer ratings of former vocational students. 
(Legislation specifies the use of employer percep- 
tions as the required empirical indicator.) 


(7) 


Methodological considerations 
(evaluation desigii, sampling 
procedures, statistical analysis): 


Sampling procedure as used in SVEA data system. 


(8) 


Evaluation standard {basis for 
interpreting outcome data to 
judge program effectiveness): 


Higher ratings for (a) former vocational students 
than for comparable workers doing the same jobs 
who did not take vocational education, or (b) the 
present cohort of vocational students as compared 
with earlier cohorts. 


(9) 


r^Ata rpni J i fprnpntQ /inQtru- 
L>u vh I w^u I i I il* 1 1 La \iiiaLiu 

ments, procedures, data base); 


\L\\\\J\\JyXz\ ft;a^Uflaca lU ci nlati aUiVtiy dl Liit; Lliiit; 

of Student follow-up. Questionnaire designed by 
SEA or LEA based on national VEDS, adapted to 
provide for comparison of vocational students 
with nonvocational, or supplemented with a 
second questionnaire to collect data on employees 
without a vocational background. 
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VOCATIONAL EDUCATION PROGRAMS Outcome No. 11 





Checklist of Elements 


Information for This Outcome* 


(10) 


Feasibility of investigating 
this outcome (conceptual, 
administrative, cost, time, 
other considerations): 


The employer ratings of former vocational stu- 
dents will be easily and inexpensively obtained 
under established VEDS requirements. Validity of 
responses may be questioned, including employer 
comparisons of vocational with nonvocational 
students. A separate survey of employer percep* 
tions of nonvocational students would be difficult 
and costly. 


(11) 


Findings (results of data 
analysis, warranted infer- 
ences, generalizability): 


High ratings given by employers to the training 
and job preparation of the employees can be 
interpreted as a positive indication of vocational 
program success, If responsei vary by item, 
school, employer, industry, or occupational 
program this can identify potential strengths and 
weaknesses. 


(12) 


Potential impact o* findings 
(implications for policy, 
program design and manage- 
ment, image of vocational 
education): 


Findings might demonstrate the need for program 
improvement and, depending on their nature, 
serve as a demonstration to the educational com- 
munity, the public, and policy makers that 
vocational education does indeed make a differ- 
ence in preparing youth for jobs. 


(13) 


Dissemination of findings and 
suggested applications: 


Share procedures, instruments, and results with 
RCU and advisory council networks, Other 
dissemination dependent on findings. 


(14) 


Implications of findings for 
further RD&E activities and 
relationship of this evalu- 
ation to previous studies: 


Since this outcome is addressed by VEDS, good 
opportunities will exist for comparing data- 
collection procedures, responses, and interpreta- 
tions of results.^ 


Notes: * See abbreviations at end of table. 
° See notes at end of table. 
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SOME KEY OUTCOMES FOR USE IN EVALUATING 



OUTCOME NO. 12 • 


AREA ECONOMIC DEVELOPMENT POTENTIAL 


F 

L 


ESSENTIAL ELEMENTS OF INFORMATION FOR AN EVALUATION STUDY 




Checklist of Elements 


Information for This Outcome* 


(1) 


Outcome question to be 
answered: 


i 

What effect does vocational education have on 
the attractiveness of the state or local communitv 
as a site for business growth and the location of 
new industry? 


(2) 


Outcome hypothesized as 
the dependent variable: 


Enhancement of the area's ootential for economic 
development. 


(3) 


Affected entity (student 
population or other entity 
in which the outcome 
is observed): 


State or local economy. (The study might also 
investigate net effects, if any, on the national 
economy as a whole.) 


(4) 


Program identified as the 
strategic independent variable: 


The existence of a substantial program of high 
school, postsecondary, and adult vocational 
education in the area. 


(5) 


Rationale for hypothesized 
outcome and appropriateness 
of the outcome as a basis for 
evaluating the program: 


Vocational education imparts occupational skills 
to men and women who, for the most part, seek 
employment with business firms in the private 
sector. Vocational programs also attempt to 
develop functional work attitudes and behaviors. 
All of these factors influence worker productivity, 
which is a basic determinant of a firm's efficiency! 
To the extent that vocational education achieves 
its objectives and also reaches large numbers of 
vyorkers, industry will find the area more attrac- 
tive as a site for investment. Vocational education 
has an appropriate interest in factors that influ- 
ence employment opportunities for its students. 


(6) 


Empirical indicators of 
the outcome: 


Statements from a cross-section of organizations 
and individuals who are knowledgeable regarding 
factors that influence industrial location and 
business investment decisions.'' 


(7) 


Methodological considerations 
(evaluation design, sampling 
procedures, statistical analysis): 


Case study design. A purposive sample of 
respondents will be drawn from business 
(especially representatives of new and recently 
expanded firms), government, labor, and academ- 
ia. Statistical analyses will be limited. 
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VOCATIONAL EDUCATION PROGRAMS 



Outcome No. 12 





Checklist of Elements 


information for This Outcome* 


(8) 


Evaluation standard (basis for 
interpreting outcome data to 
judge program effectiveness): 


Favorable appraisal of the influence of 
vocational programs on economic development 
potential. In the absence of systematic data 
collection and a rigorous comparison design, the 
most credible evaluation conclusions might be th 
appraisals of relatively objective experts having 
little or no stake in the nature of the judgments. 


(9) 


Data requirements (instru- 
ments, procedures, data base): 


Data describing current and past vocational 
education program effort; instruments for inter- 
viewing or conducting mail surveys to obtain 
respondents' views on the impact of vocational 
education. 


(10) 


Feasibility of investigating 
this outcome (conceptual, 
administrative, cost, time, 
other considerations): 


Given (a) the case study design, (b) reliance on 
judgmental data, and (c) designation of potential 
ramer than actual economic development, this 
study is quite feasible, However, the findings may 
hot be credible; yet a more ambitious design 
might produce results of even lower credibility. 


(11) 


Findings (results of data 
analysis, warranted infer- 
ences, generalizability): 


Findings, whether positive or negative, would add 
to our knowledge of the effects of vocational 
education on an entity other than individual 
students, thereby increasing awareness of the 
range and diversity of possible outcomes. 


(12) 


Potential impact of findings 
(implications for policy, 
program design and manage- 
ment, image of vocational 
education): 


Positive findings could enhance the reputation 
of vocational education as a program that con- • 
.ributes economic benefits to the area. Other 
states and local areas might want to re-examine 
their own vocational programs in the context of 
economic development. Care should be taken not 
to claim that benefits accruing to a particular area 
are necessarily benefits for the nation as a whole, 


(13) 


Dissemination of findings 
and suggested applications: 


Instruments, procedures, and results should be 
shared with evaluators in other LEAs and SEAs. 
Audiences for further dissemination dependent 
on the nature of the findings. 


(14) 


Implications of findings for 
further RD&E activities and 
relationship of this evalu- 
ation to previous studies: 


Since this study is basically exploratory, it could 
generate a number of testable hypotheses for 
future research. Innovative procedures and modi- 
fied inst.uments should be developed. Emphasis^ 


iMotes: * See abbreviations at end of table. 
^ See notes at end of table. 
^ Continued at end of table. 
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Table 4. 



SOME KEY OUTCOMES FOR USE IN EVALUATING 



OUTCOME NO. 13 • IMPROVING THE OCCUPATIONAL DISTRIBUTION 

OF MINORITY WORKERS 


1 


ESSENTIAL ELEMENTS OF INFORMATION FOR AN EVALUATION STUDY 




Checklist of Elements 


Information for This Outcome* 


(1) 


Outcome question to be 
answered: 


What effect does the existence of an extensive 
vocational education program have on the 
occupational distribution of minority workers? 


(2) 


Outcoriofiiiypothesized as 
the dependent variable: / 


Improvement of the occupational distribution of 
minority workers. 


(3) 


Affected entity {student 
population or other entity ^ \. 
in which the outcome 
is observed): 


V Employed 'abor force in a particular labor r 
— market area. 


(4) 


Program identified as the 
strategic indapendent variable: 


Existence for^more than three years of a sub- 
stantial program^ vocational education, 
preferably at all levels of training (s£v>ui.dary, 
postsecondary, and adult). 


(5) 


Rationale for hypothesized 
outcome and appropriateness 
of the outcome as a basis for 
evaluating the program: 


To the extent that minority students enroll in 
vocatiorial programs and find emoloyment in 
occupations related to their traininq, changes in 
the occupational distribution of employment 
should reflect the pattern of minority participa- 
tion in vocational training. It is a goal of voca- 
tional education and natioiial policy to increase 
the access of minority workers to a wider range 
of occupations." 


(6) 


Empirical indicators of 
the outcome: 


Distribution of (a) vocational enrollments, (b) 
placements, and (c) employed workers, by 
occupation and race/ethnicity. ■ 


(7) 


Methodological considerations 
(evaluation design, sampling 
(jrut-cijures, siaiiSTicai analysis;: 


Case study design. Select one or more labor 
market aiuiis ihji draw a high proportion of 
workers from an educational system having a 
large, wp!! p-tr.iblished vocational program. 
Descriptive statistics showing occupational 
distrioutir i of minority and other workers. 


(8) 


Evaluation standard (basis for 
interpreting outcome data to 
judge program effectiveness): 

1 


Ratio' of minority-to-total enrollment and place- 
ments higher than minority-tc-total employment 
would be an indication that vocational education 
is helping move minority workers toward greater 
job equality. Comparisons can also be made 
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VOCATilONAL EDUCATION PROGRAMS 



Outcome No. 13 





Checklist of Elements 


Information for This Outcome* 


(8) cc 


ntinued 


betvveen labor market areas having similar demo- 
gra^ihic and economic characteristics but differing 
in the extent of their vocational programs. Before- 
and-after comparisons within a given area are 
possible but would not likely yield valid results 
since so m?»ny factors besides vocational programs 
affect employment patterns over time. 


(9) 


Data requirements (instru- 
ments, procedures, data base): 


Vocational enrollments, job placements, and 
existing labor force statistics showing the distribu- 
tion of employed workers by occupation and 
race/ethnicity. 




redsiDiiiiy ot invesiigaiing 
this outcome (conceptual, 
administrative, cost, time, 
other considerations): 


witn Its use ot tne case stuay aesfgn, this inquiry 
appears highly feasible. Existing data can be used. 
Costs of analysis will be low. Credibility, however, 
may be questionable, depending on how skillfully 
the data are represented and the comparison made 
with the "control" area. 


(11) 


Kindings (results of data 
analysis, warranted infer- 
ences, generalizability): 


Positive findings would demonstrate significant 
-labor market benefits accruing to minority 
workers. This would show that vocational educa- 
tion is an effective mecha'^i >m for advancing 
national policy. 


(12) 


Potential impact of findings 
(implications for policy, 
program design and manage- 
ment, image of vocational 
education) : 


Findings described in (11) would counter the 
allegation that vocational programs fail to serve 
the needs of minority groups. More minority 
students might be attracted to vocational educa- 
tion at secondary, postsecondary, and adult levels. 


(13) 


Dissemination of findings 
and suggested applications: 


Methodology and findings should be shared with 
RCUs and other evaluators. Findings could be 
reported in local news media as well as through 
appropriate national channels. 


(14) 


Implications of findings for 
further RD&E activities and 
reiaiionsn p ot inis evalu- 
ation to previous studies: 

. J 


The study should be critiqued and repeated for 
the "program" area, with variations that address 
aspects OT ine siuay xnai were criticizeo. bimiiar 
studies should be done in other labor market 
areas. Where significant differences exisc, alterna- 
tive hypotheses (other than the existence of 
vocational programs) should be formulated and 
tested for their explanatory povver.*^ 


Notes 


* See abbreviations at end of table. 
See noteb at end of table. 



Table 4. 



SOME KEY OUTCOMES FOR USE IN EVALUATING 



OUTCOME NO. 14 


• TRAINING-RELATED EMPLOYMENT 


E 


SSENTIAL ELEMENTS OF- INFORMATION FOR AN EVALUATION STUDY 




Checklist of Elements 


Information for This Outcome* 


(1) 


Outcome ciuestion to be 
dnswered' 


To what extent do former high school vocational 
students find employment in occupations related 
to their training? 


(2) 


Outcome hypothesized as 
the dependent variable; 


Training-related employment. 


(3) 


Affected entity (student 
population or other entity 
in which the outcome 
is observed); 


Former high school students who completed 50 
percent or more of an approved high school 
vocational program. 


(4) 


Program identified as the 
strategic independent variable: 


Completion of 50 percent or more of an approved 
high school vocational program. 


(5) 

- 


Rationale for hypothesized 
outcome and appropriateness 
of the outcome as a basis for 
evaluating the program: 

/ 


Vocational programs purportedly are designed to 
impart entry-level skills in specific occupational 
areas. It is assumed that possession of these skills 
will enhance a person's employability in that 
particular occupation To the extent that students 
find jobs in occupations unrelated to their skill 
training, or do not find jobs at afi, it is felt that 
the training was to some extent wasted. The 
Education Amendments of 1976 specify that 
states shall evaluate their vocational programs in 
part "according to the extent to which program 
completers and leavers . . . find employment in 
occupfations related to their training." 


(6) 


hmpirical indicators of 
the outcome: 


Labor force status of the former students as of a 
designated time period (e.g., survey week) 
approximately eight months after the end of the 
preceding school year; and occupational attach- 
ment of those former students who were observer' 
to be employed as of the survey period. 




MethodoloQicdl consideiutions 
(nvttluation design, sam()ling 
pro. »fiures. Statistical annlysis); 


Case-study design. Random samples of former 
students m several different vocational program 
are^s. Frequency distribution of individuals wlio 
are not omfjloyod (by reason, such as continuing 
their schooling on a full-time basis), are om()lnyed 
in a training related occupation, or arp emfjloyeci 
in .m unrelated occijf)ation. 



•5H 
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VOCATIONAL EDUCATION PROGRAMS 



Outcome No. 14 





Checklist of Elements 


Information for This Outcome* 




CvdIUalluri dianUdru \DaSlb TOi 

interpreting outcome data to 
judge program effectiveness): 


No quantitative standard seems appropriate for 
application to the program population overall. 
Interpretation of outcome data will be judgment- 
al. However, comparisons can be made among 
particular occupational areas by sex, race, and 
category (completers vs. early leavers). 


(9) 


Data requirements {instru- 
ments, procedures, data base): 


Background and employment data will be required 
on program students, as described in (6) above. 


no) 


FpA^ihilitvy r^f i n v/oct inrjf 1 nn 

rcuoiuitiiy ui niVcbiiyaiiriy 

this outcome (conceptual, 
administrative, cost, time, 
other considerations): 


oince Toiiow-up oata are routinely collected, this 
outcome appears highly feasible. Two major 
problems do exist however. One is the basis for 
allocating students to the "training-related" and 
"unrelated" categories. The other relates to the 
lack of a quantitative evaluation standard. 


(11) 


Findings (results of data 
ciriciiybib, wdrrdiiicu inTci" 
ences, generalizability): 


Relatively "high" percentages of training-related 
employment would suggest that specific occupa- 
tional training is put to effective use, while "low" 
percentages might imply the opposite. Differences 
among particular program areas could invite re- 
examinaw.^ii of the skill content and relevance of 
the curricula. 




r oteniiai impact ot Tinuings 
(implications for policy, 
program design ^qd manage- 
ment, image of vocational 
education): 


To the extent that high rates of training-related 
placement are valued, programs could be revised 
to incorporate features (e.g., employer participa- 
tion in curriculum design) associated with the 
high rates. Resources and enrollment might be 
allocated among program areas. 


(11) 


L/lbbuITi 1 rid llUr! Of IlriUinyS 

and suggested applications: 


Teachers, students, counselors, and parents should 
be i:"formecJ of reported placement rates by occu- 
p * ^ijlds. Instruments, procedures, and 
resu ^oould be shared with evaluators in other 
^ l.E As and SEAs. Other dissemination to targeted 
audiences as appropriate. 


(14) 


Implications of findings for 
further ^DikB activities and 
relationship of this evalu 
ation to previous studies: 


The study should be replicated find expanded to 
include additional program areas, school systems, 
and labor markets. Placements could be compared 
not only with enrollments and completions but 
also with the number of students who actually '' 


N<ites * Snf? ab[)reviations at (?rKl of t;ihlp 
Continued nt nnd of table. 
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Table 4. SOME KEY OUTCOMES FOR USE IN EVALUATING 



OUTCOME NO. 15 • DEVELOPMENT OF CONSUMER SELF-HELP SKILLS 


E 


SSENTIAL ELEMENTS OF INFORMATION FOR AN EVALUATION STUDY 




Checklist of Elements 


lnforma'''on for This Outcome* 


(1) 


Outcome question to be 
answered: 


To what extent do high school vocational students 
acquire economically valuable consumer self-help 
skills? 


(2) 


Outcome hypothesized as 
the dependent variable: 


Development of consumer self-help skills. 


(3) 


Affected entity (student 
population or other entity 
in which the outcome 
is observed): 


Former students who completed one semester 
or more of an approved high school vocational 
program. 


(4) 


Program identified as the 
strategic independent variable: 


Completion of one semester or more of an 
approved high school vocational program. 


(5) 


Rationale for hypothesized 
outcome and appropriateness 
of the outcome as a basis for 
evaluating the program: 


Vocational programs, including Consumer Home- 
making courses, impart knowledge and develop 
skills that are useful not only on the job but also 
at home. To the extent that former vocational 
students can apply these skills in the production 
of goods or services for the benefit of their own 
households, they have economic value "in use" if 
not "in exchange." The dollar value of this pro- 
duction can be estimated on the basis of what the 
goods or services would cost if they were pur- 
chased in the market. 


(6) 


Empirical indicators 
of the outcome: 


Reports from a survey of former vocational stu- 
dents (confirmed by observation or third-party 
testimony where feasible) on actual use of skiHs 
derived from vocational programs. Estimates of 
the market value of goods and services produced 
for household consumption with these skills. 


(7) 


Methodological considerations 
(evaluation design, sampling 
procedures, statistical analysis): 


Case-study design. Stratified random samples of 
former students for several vocational program 
areas, by year they left school. Statistical analyses 
of the dollar value of nonmarket production. 


(8) 


Evaluation standard (basis for 
interpreting outcome data to 
judge program effectiveness): 


Focusing on a sample of distinctive vocational 
skills such as automotive mechanics, carpentry, 
sewing, and electrical arts would make it plaus 
ible to assume that vocational students have a 
clear advantage over nonvocational students in 
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VOCATIONAL EDUCATION PROGRAMS 



Outcome No. 15 





Checkljst of Elements 


Information for This Outcome* 


!3) c< 


jntinued 


their ability to perform related self-help tasks. 
The economic value of such skills could be 
estimated at some reasonable percentage of 
market value of the goods or services. 


(9) 


Data requirements (instru- 
ments, procedures, data base): 


Background data would be required on the 
vocational students, the courses they took, and 
uses they made of school -derived vocational 
skills. Data could be collected in their senior 
year, one year later, and then four years after 
that. In order to complete the study in a shorter 
period of lime, data could be collected retro- 
spectively rather than on a longitudinal basis over 
the five-year period indicated. 


(10) 


Feasibility of investigating 
this outcome (conceptual, 
administrative, cost, time, 
other consideratlens): 


As a naive-model case study design, this would be 
feasible at least as an exploratory study or to 
illustrate specific nonmarket payouts by means 
of personalized case studies. The cost of inter- 
views would be high. Determining the source of 
a skill (e.g., school-based vocational program vs. 
informal learning at home) would be difficult, It 
does not appear that a satisfactory study could 
be carried out consistent with classical experi- 
mental or quasi-experimental designs. 


(11) 


Findings (results of data 
analysis, warranted infer- 
ences, generalizability); 


If it is found that former vocational students 
possess school-derived skills and use them to 
produce economically valuable goods and 
services for the direct benefit of their own house- 
holds, this fact will demonstrate a nonmarket 
("practical arts") outcome that could expand 
the framework used for judging vocational educa- 
tion's value. At the present time, vocational 
education outcomes having nonmarket value are 
a::sociated primarily with the field of Consumer 
Homemaking. Regarding other program areas, to 
the exte-'t that completers and leavers are not 
placed in ^obs related to their training, vocational 
education might be seen as failing in its mission, 


(12) 

. . 1 


Potential impact of findings 
(implications for policy, 
program design and manage- 
ment, imag'i of vucalional 
education): 


While positive findings could increase public 
appreciation of vocational education's value, they 
could also create potential conflict. More students 
might desiro to eriroll in vocational education in 
order to acquire the seif-help skills but not to 
prepare for ontry-levei t^mployment in the partic 
ular occupational area. 
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VOCATIONAL EDUCATION PROGRAMS Outcoma No. 15 (continued) 





Checklist of Elements 


- 

Information for This Outcome* 


(13) 


Dissemination of findings 
and suggested applications: 


Instruments, procedures, and results should be 
shared with evaluators in other LE As and SE As. 
Other disseminavion as appropriate. 


(14) 


Implications of findings for 
further RD&E activities and 
relationship of this evaluation 
to previous studies: 


Since this is essentially an exploratory study, 
other SEAs and LEAs should be encouraged to 
vary the procedures and focus on different 
vocational programs. A similar study could also 
be done at the postsecondary level, although it is 
generally assumed that occupational education 
is more closely tied to expectations for employ- 
ment in the field of training. Illustrative case 
studies could be used In developing benefit-cost 
analyses. Broader survey data might be collected 
In collaboration with agencies such as the 
Cooperative Extension Services." 



Notes: * See abbreviations at end of table. 
^ See notes at end of table. 

; 
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Table 4. SOME KEY OUTCOMES FOR USE IN EVALUATING 
VOCATIONAL EDUCATION PROGRAMS (continued) 



Abbreviations: 


CETA 




Comprehensive Employment and Training Act 


LEA 




Local Education Agency 


NAC\AE 




National Advisory Council on Vocational Education 


NLS 




National Longftudinal Surveys (Ohio State University) 


RCU 




R^^arch Coordinating Unit 


RD&E 




Research, Development, and Evaluation 


SACVJE 




State Advisory Council on Vocational Education 


SEA 




State Education Agency 


SVEA 




State Vocational Education Agency 


VEDS 




Vocational Education Data System (National Center for Education Statistics. U.S. 






Department of Education) ' 


VSO 




Vocational Student Organization 



Notes: (Numbef^ in parentheses refer to outcome numbers.) 



( j)a Literature references include Leonard A. Lecht, Evaluating Vocational Education: 

Policies and Plans for tho 1970s (New York: Praeger. 1974); and National Assessment of 
Educational Progress, Career and Occupational Development Objectives (Denver: 
National Assessment of Educational Progress. Education Commission of the States. 
1971). None of the thirty-one empirical studies reviewed in Bolland, Evaluative 
Bibliography, cited in Chapter I above, address this outcome. 

(2)a None of the empirical studies reviewed in the Evaluative Bibliography used 

performance testing techniques to investigate the extent to which vocational students 
developed useful occupational skills. However, several studies, including one by the 
Texas Advisory Council for Vocational Education attempted to assess employer 
satisfaction with the training and preparation of former vocational students. On this 
general topic see James R. Sanders and Thomas P. Sachse, Editors, Problems and 
Potentials of Applied Performance Testing (Portland: Northwest Regional Education 
Laboratory. December 1975); and Janet E. Spirer, Performance Testing: Issues Facing 
Vocational Education (Columbus: National Center for Research in Vocational 
[Iducation. The Ohio State University. 1980). 

{3)a See John T. Grasso and John R. Shea. Vocational Education and Training: Impact on 
Youth (Eierkeley: Carnegie Council on Policy Studies in Higher Education. 1979): and 
Robert H, Meyer and David A. Wise, hiigh School Preparation and Early Labor Market 
ExporioncQ, Working Paoer No. 342 (Cambridge: National Bureau of Economic 
Research, May 1979) Studies reviewed in the Evaluative Bibliography do not yield 
cles'' findings. Neither the Carnegie nor NBER studies cited hero indicate clear 
differences m unemployment rules based on high school curriculum. 
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(4) a In support of the claim that "knowledge of the world-of-work" pays dividends in the 

labor market, see Herbert S. Parnes, ot al.. Career ThrQsholds, Vol. 7, Manpower 
Research Monograph No. 16 (Washington: U.S. Department of Labor, 1970). Other 
studies document the production of world-of-work knowledge but do not find that such 
knowledge leads to higher wages. See Richard V. Kauffman, A Study of tho 
Educational Production Function (Ph.D. Dissertation, Department of Economics, 
Colorado State University; hard copy published on demand by University Microfilms, 
Ann Arbor. 1974). 

(5) a (continued) 

from personal investment in schooling would have special appeal to economically 
disadvantaged youth, for whom high school completion plus two or four years of 
postsecondary education might seem excessive time to spend before beginning to earn 
a living. Because of their background and experience, the students may find vocational 
subjects easier and more interesting than academic courses. This feeling could 
encourage them to attend schdo\ more regularly, study harder, and slay in school 
rather than drop oilt. Vocational programs often provide opportunities for students to 
earn money while still in school, thereby further reinforcing their commitment to 
complete high school. Educators, employers, and the general public all value high 
school completion. 

(5) b For a brief literature review and a multiple regression analysis of factors associated 

with school retention, see Robert L. Ellison and David G. Fox, Biographical and 
Academic Correlates of High School Completion (Raleigh: North Carolina Department 
Public Instruction, 1973). Also see Jerald G. Bachman et al.. Youth in Transition, 
Volume III: Dropping Out— Problem or Symptom? (Ann Arbor: Institute for Social 
Research, University of Michigan, 1971). 

(6) a Literature references include Dennis R. Collins, An Assessment of Benefits Derived from 

/Membership in a Vocational Student Organization in the Vocational, Technical and 
Adult Education System (Menomonie, Wisconsin: Center for Vocational, Technical, and 
Adult Education; University of Wisconsin— Stout, August 1977); James C. Nance, 
National Institute on the Rolo of Youth Organizations in Vocational Education (Final 
Report, DHEW project 16. 9-0322, Trenton, NJ, August 1971) and A. A. Clark, An 
Analysis of Leadership and Self-Confidfince and /or Self -Acceptance Outcomes from 
Student Participation in Distributive Education Clubs of America (Ph.D. Dissertation 
University of Minnesota. 1978). 

(7ia It IS sometimes argued that vocational students are at a disadvantage when e;itering a. 
baccalaureate program, though they may enjoy advantages over studer.ts who did not 
take a vocational program when they choose a two-year occupational program. The 
concern in the evaluation study is not with the possible effect of high school vocational 
education on motivation for postsecondary schooling but rather with access to and 
success in various types of such programs. 

(^b See Davids Bushnell (Editor), Help Wanted- Articulating Occupational Education at 
the Post-SQcondary Level (Columbus: Center for Vocational Education, Ohio State 
L/niversity. 1977). and Leonard A Lecht. Evaluating Vocational Education: Policies and 
Plans for the 1970s (New York Praeger. 1974) pp. 84 ff 
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(8) a Several empirical studies of earnings are reported in Bolland. Evaluative Bibliography. 

Also see William J. Conroy, Jr.. "Some Historical Effects of Vocational Education at the 
Secondary Level," Phi Delta Kappen. Vol. 61. No. 4 (December 1979) pp. 267-271; and 
Grasso and Shea. Vocatior)al Education and Training: Impact on Youth (6p. cit.). 

\ 

(9) a According to Coburn, Salem, and Mushkin (1973). educational outcomes c«\n be 

categorized in terms of investment and consumption characteristics. Investrrvent-type 
outcomes refer to effects of education on income and employment. Consume^r-type 
outcomes refer to the direct satisfaction of wants that students (and others ini>olved in 
education) experience as a result of their participation in educational program^. 
Earning higher wages by using occupational skills on the Job is an investment-type 
outcome; feeling happy about attending vocational courses as opposed to acadehnic 
classes is a consumption-type outcome. 

(9) b See Christopher Jencks et al., Inequality: A Reassessment of the Effect of Family and 

Schooling in America (New York: Basic Books, 1972); and Grasso and Shea, op cit 
pp. 14 ff. 

(10) a For a description of the National Longitudinal Surveys see Taylor. Darcy. and Bolland, 

Vocational Education Outcomes, op. cit., pp. 24f. 

(10) b References include Gerald G. Somers. The Effectiveness of Vocational and Technical 

Programs (Madison: Center for Studies in Vocational and Technical Education. 
University of Wisconsin, 1971); Jobseeking Methods Used by American Workers 
(Washington: U.S. Department of Labor. Bureau of Labor Statistics, 1975); and Robert 
G. Wegmann, "Job-Search Assistance Programs; Implications for the Schools." Phi 
Delta Kappan, Vol. 61, No. 4 (Decemoer 1979) pp 271 ff. 

(11) See Larry L. Smiley, Employer Satisfaction with the Skills of Vocational Education 
Graduates in North Dakota (Grand Forks: Bureau of Educational Research and 
Services, University of North Dakota, August 1976); Martin Hamburger and Harry E. 
Wolfson. 1000 Employers Look at Occupational Education (New York: Board of 
Education of the City of New York, July 1969); The Adequacy of Vocational and 
Technical Education (Columbia: South Carolina Advisory Council on Vocational 
Education. February 1976); and A Program Review of Secondary Vocational Education 
m Ohio: Job Placement and State Funding, Staff Report No. 126 (Columbus: Ohio 
Legislative Services Commission. April 1978). All four of the studies are reviewed in 
Bolland, Evaluative Bibliography. 

(12) a (continued) 

■Judgmental" information of this type may seem less desirable than data on actual 
investment, growth rates, and labor productivity. The latter variables, however, are 
influenced by so many factors that it would be extremely tenuous to attribute causation 
to the existence of vocational programs. Perceptions of experts might therefore be 
more valid and credible. 

(I2)b (continued) 

might be placed on particular occupational specialties having the greatest impact on 
industrial expansion rather than attempting to show impact for vocational education 
across the board. The capability of the vocational education system to provide quick 
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(I3)a 

(13) b 

(14) a 

(15) a 



retraining and skill upgrading for adults might be worthy of special study. References 
include Paul V. Braden and Krishan K. Paul, The Role of Vocational Education in the 
Nation's Bconomic Development, Information Series No. 150 (Columbus; National 
Center for Research in Vocational Education, 1979); and Don C. Garrison, "Local 
Articulation Lffort: The Tri-County Technical College," pp 11-20 in David S. Bushnell, 
Editor. Help Wanted, op. cit. 

9 

On the job aspirations and vocational education experiences of black people, see 
Ferman Moody. "The History of Blacks in Vocational Education," and related articles in 
VocEd, Vol. 55. No. I (January 1980). 

For general perspective and some historical data on the occupational distribution of 
black employment, see Andrevy/ F. Brimmer. "Economic Growth and Employment and 
Income Trends Among Black Americans." in Jobs for Americans edited by Ell Ginzberg 
(Englewood Cliffs. N.J.: Prentice-Hall, 1976). Also see Richard^Butler and James J. 
Heckman. 'The Government's Impact on the Labor Market Status of Black Americans: 
A Critical Review," in Equal Rights and Industrial Relations, edited by L.J. Hausman et 
al. (Madison, Wisconsin: Industrial Relations Research Association. 1977). 

(continued) 

seek jobs in training-related areas. Instruments and procedures for determining 
employment status and training-relatedness should be further developed. The entire 
retionale for valuing training-relatedness should be reviewed. References include J. 
Kenneth Little, Review and Synthesis of Research on the Placement and Follow-up of 
Vocational Education Students (Columbus; Center for Vocational and Technical 
Education, The Ohio State University, 1970); and The Status of Vocational Education: 
School Year 1975-1976 (Columbus: National Center for Research in Vocational 
Education, The Ohio State University. 1978). A number of the outcome studies reviewed 
in Bolland, Evaluative Bibliography, address the subject of training-related job 
placement. 

Literature references include Kathryn E. Walker and Margaret E. Woods. Time Use: A 
Measure of Household Production of Family Goods and Services (Washington. D.C.; 
American Home Economics Association. 1976); and Kathryn E. Waiker and William H. 
Gauger. The Dollar Value of Household Work, Information Bulletin 60. Consumer 
Economics and Public Policy No. 5 (Ithaca: Cooperative Extension Services. Now York 
State College of Human Ecology, 1973). 
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IV. SOME RESULTS FROM PILOT TESTING 



Efforts to develop operational procedures for evaluating vocational education on the basis of 
outcomes included some pilot testing with existing data available from state education agencies. 
The two states involved in the pilot tests will be referred to in this report as State-A and State-B. 



In consultation with vocational education officials of the two cooperating states, the project 
staff selected for testing "Reducing the Risk of Unemployment for Minority Youth" (Outcome No. 
3 in Table I above). 

Both states involved in the tests have significant minority populations, both regard the 
general area of labor market outconnes to be highly important for judging the effectiveness of 
vocational education, and both had the necessary data on vocational students available in their 
computerized management information systems. Neither the project staff nor the respective 
states consider this particular labor market outcome to be the most appropriate single criterion 
for evaluating vocational programs. However, studying this outcome offered certain advantages, 
including the face*value importance of the outcome, feasibility in terms of data availability, and 
conceptual clarity. Moreover, if the rtudy could be carried out successfully for the subset of 
minority students, it Vi^ould be relatively simple to apply the same procedures to other subsets or 
to vocational completers in toto. 

One reason the pilot test focused on minority completers rather than all completers^as to 
demonstrate vocational education's sensitivity to the plight of young minority workers, for whom 



national unemployment rates run— depending on age, race/ethnicity, and place of residence— as 
high as 30-40%. It should be pointed out that the outcome was defined in terms of reducing the 
risk of unemployment for minority vocational completers, not the effect of vocational education 
on the state's or nation's aggregate minority youth unemployment rate. 



B. Findings and Procedures 

Table 5 summarizes data for State-A on program completers, sample size, completers 
actively participating in the labor force during the follow-up survey period, and unemployment 
rates calculated for minority completers and all completers. The calculated unemployment rate 
as of early 1979 for minority completers in State-A was 8.9%. Data for 1978 (not discussed in this 
report) indicated an unemployment rate of 12.2%. 

Table 6 reports similar data for Stato-B, where only one year's statistics were available. The 
calculated unemployment rate for minority1::ompleters as of early 1979 was 9.8%. It should be 
noted that the racial/ethnic composition of State-B's minority population differs markedly from 
that of State-A. dates of the employment-status survey period differed slightly, and the follow-up 
survey in State-B included 100% of the previous year's completers rather than a sample of the 
completers as in State-A 



A. Outcome Selected for Testing 
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TABLE 5 

UNEMPLOYMtNT RATES IN EARLY 1979, STATE-A HIGH SCHOOL 
VOCATIONAL COMPLETERS, MINORITY AND ALL COMPLETERS 



00 



r — 

Student 
Category 


1977-7E 


i 


Status as of Early 1979-^ 


Population 


Sample 


Unknown 


Not Available 
for Placement . 


Available for Placement 


Unemployment 
Rate'* 


Employed 3 Unemployed 


Minority Completers 


2,110 


1,070 
(100%) 


111 
(10.4%) 


319 

(29.8%) 


583 57 
(54.5%) (5.3%) 


8.9% 5 


All Completers 


12,490 


7,133 
(100%) 


697 
(9.8%) 


2,225 
(31.2%) 


4,001 209 
(56.1%) (2.9%) 


5.0% 5 



NOTES. 

1^ Based on statewide follow-up survey conducted during the period January 1 - February 15, 1979 
2/ Includes persons enrolled in school and other. 

3/ Includes persons employed in training-related jobs as well as those employed In jobs not related to the field of their vocational training. 

4/ Vhe unemployment rate (UR) is calculated by dividing the number reported as unemployed (U) by the sum of employed (E) plus the unemployed (UR _ U 
This is the method used by the U.S. Bureau of Labor Statistics (BLS) to calculate unemployment rates. E^U 

h Cofifiiiefice intervals were calculated indicating a 95% probability that actual unemploy ment rates we re betweert 6.7% and 1 1.1% for minority completers and 
between 4 3% and 5.6% for all completers. The formula used was p + ^ l /p (i p) 



.025 



n 1 



SOURCE Stale A vocational education agency, management information system. 
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TABLE 6 

UNtMPLOYMENT RATES IN EARLY 1979, STATE-B HIGH SCHOOL VOCATIONAL COMPLETERS: 

BLACK, WHITE, AND ALL COMPLETERS 





— 


Status as of Early 1979 ^ 


Student Category 


Number of 
Complaters 
in 1977-78 


Unknown 


Not Available 
for Placement ^ 


Available for Placement 
Employed ^ Unemployed 


Unemployment 
Rate 4 

K 
1 


Completers 

i 


8,283 
1100%) 


790 
(9.5%) 


3,884 
(46.9) 


3,256 353 
(39 3) (4.3%) 


— — j 

9.8% 5 j 

1 


A'hite Completers 


10,. 88 
(100%) 


1,333 
(7.3%) 


5,059 
(27,8%) 


11,420 376 
(62.8%) (2.1%) 


3.2% ^ 


All ComplHters 


26,480 
(100%) 


2,126 
(8.0%) 


8,948 
(33.8%) 


14,677 729 
(55.4%) (2.8%) 


4.7% ^ 



NOTTS 

P B<jsf?(i nn stut^^wic^^^ foilow i.p survey conciucted during the period February 1 • March 15. 1979. 
2/ Includes persons enrt^llod »n school and other. 

3/ lfu.-lij«fps person^. f?mpU)yeii in trdinmq related jobs as well those employed in jobs not related to the field of their vocationdl training. 

4 Tfie unpfTiploympni raff? (UR) is calculated by dividinq the number reportotl c's unemployed (U^ by the <»iim nf nmpioynti (E) ()lus the 
un*?f^ipl()yed;ijR LL' This is the method that the U.S. Bureau of Labor Statistics (BLS) uses to calculate unemployment rates. 

[- 

5/ Confuicnce intervals were calculated indicating that there is a 95% probability that the actual unemployment rates were between 8.8". 
anU 10 8% for black completers, bet ween 2.9 % and 3.5% for white compl(^t.*rs. an<J between 4.4% and 5.1% for alt completers. The 
formula used was p ♦ / 1 /p( 1 p) 

n 1 



02^1 



SOUHCt. !Statf H vocatiot^al tnlucatjur^ .iqency. manageaient infortnatior) systHm. 
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It must be emphasized thai the mam purpose of this section is to illustrate how certain 
evaluation procedures can be applied to a particular outcome and to highlight some problems 
encountered, not to report in depth on empirical evaluation studies per se. No attempt was made 
to design identical studies for the two states. Nor is there the opportunity to provide detailed 
information on how the two states differ in terms of their vocational programs, data systems, or 
the specifications and procedures used in conducting this particular outcome study. Most of th^ 
following discussion focuses on State-A. where the pilot testing first occurred and where more 
staff time was allocated to the study. 

As in<?cated in Table 3 above, fourteen essential elements of information are identified for an 
evaluation study. How this checklist of eleme.-: 3 can be applied to the risk-of-unemployment 
outcome is illustrated in general terms in Table 4, Outcome No. 3. For the specific case of 
Stato-A, details are given hero only for those checklist items where the general information is 
inaoequate 

In State-A. the 'affected entity" consisted of vocational completers identifie'1 in school 
reports as American Indian,- Black (not of Hispanic origin), Oriental, or Hispanic ("Latin- 
American, Spanish Surnamed American, or Mexican-American"— sometimes termed "Chicano"] 
In 1977-78, Hispanics accounted for about 70% of minority completers, while minority students in 
turn accounted for 16.9% of all completers. 

The prograrr. or -treatment" identified ss the strategic independent variable was completion 
during 1977-78 of a high school vocational program approved under the state's vocational 
education act. A basic requirement for approval, entailing financial support from the state, is that 
the program "be designed to provide students vith a.i entry level occupational skill." Excluded 
from the list of state-approved programs is the area of consumer homemaking. 

As the entries for Outcome No. 3 in Table 4 indicate (see checklist Items 7 and 8), the 
suggested evaluation design calls for comparing the calculated unemployment rate for minority 
vocational completers with one or more similar groups of labor force participants. In the context 
of the State-A evaluation study, the most relevant characteristics for achieving similarity of 
qroups included age. race/ethnicity. and level of schooling (i.e., high school graduates versus 
vV(;rkers '.vith less than a high school diploma). Two other variables, labor market area and sex, 
were acknowledged to influence unemployment rates but assumed to be partially "controlled for" 
so long as comparison group'j were limited to the State-A labor force. Data requirements 
(cht^cklisf Itf-m 9) '.herefore included labor *orce statistics not only for the designated student 
population but also for one or moro, comparison groups. 

idoany w^^at was required was an unemployment rate for State-A minority workers, age 19. 
ri if rr nil ■ whom were .'igh school graduates but Jjad no* taken vocational education. (The 
i-sfrM .f.op nf vocational completers t)y 'high school graduate" versus "dropouf was not known 
.n Sfatr-A rut state staff assumed vhat about 85% of all completers were high school graduates ) 
f-urthermorp the data on lat^or force status should reflect the same time period (the vocational 
'.)ii.)w-uD sufvPY oorioci was January-February. •.979) and the same definitions and survey 
••'>f'?ht"js 

• u jr.,. s s.;^pMSf^ • if-ai companion data did .not ovist In collaboration w.th tho Statf> 
1 .,<.,,Mt Sp; •..'■ry Af.i> -u y iSF'.Ai v^f wt'rp af)!(' to qfMU^ratf^ OStirriHtOd StatPWKjt- 

• .."ip; ,'noi)t rafon (ijfKi fr;f 

.{■ .V- wufk^.is ;mm_ :]y^>..^^ of fPhruary ^O-'q the rate was -3 7% for 1978 annual 
<v^»ra'j>" 1 



/■ 



#2' 



(b) alt minority workers (URj^ • 6 1% as of 1978, an5(ual average; data not available for 
early 1979) 

(c) all youth, age i6-24 (UR^ = 11.8% as of 1977. annual average; data not available for 1978 
or early 1979) 

(d) all rnirionty youth, age 16-24 (UR^^ - •16.4%" in 1976 and 1977. annual average; data not 
available for 1978 or early 1979. quotation marks indicate data are not of publishable 
quality). 

We also had. from our own analysis of state vocational education data, an estimated 
unemployment rate for all vocational completers as of the time period covered by the survey 

(e) all 1977-78 vocational completers (UR^ '~ 5.0%). 



And by manipulating national data published by the U.S. Bureau of Labor Statistics (BLS) in 
a variety of different reports, the staff was able to generate estimated unemployment rates as of 
various recent tirne periods for 

(1) black youth (16-24) 

(If) Hispanic youth (16-24) 

(ill) all youth (16-24) 

(ivithe teenage (16-19) subsets of black. Hispanic, and all youth 

(v) the 18 and l9-year-old subsets of black youth and aH youth 

(vi) high school graduates versus workers with less than a high school diploma (for two 
dif^f^rent age categories) 

Then, m late 1979. BLS published Special Labor Force Report 223 summarizing results of a 
special survey of the high school class of 1978. Based on supplementary questions in the October 
1978 Current Population Survey conducted by the U.S. Bureau of Census, this study reported the 
following estimated unemployment rates for the national sample of 1978 high school graduates 
who wt'-^re not onrollf^d in college 
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The virtue of the BUS estimates is that they refer to the sarTie higK school cohort as the state 
data on vocational completers, thereby controlling for age and (approximately) for educational 
attainment. Use will be made of these data below. 

it should be noted that national data consistently show unemployment rates for Hispanic 
youth to be higher than rates for all youth but substantially below the rates for black youth. The 
importance of UR differentials between blacks and Hispanics (there are also differences for 
Amencan Indians and Asians) arises from the fact that the "minority" population of one state 
may be composed of quite a different racial/ethnic mix than the minority population of another 
state of the nation as a whole. In the case of State-A, blacks account for less than 20% of the 
minority population, whereas they account for 99% of the minority population in State-B. 
Nationally, blacks constitute 92% of the labor force category designated "Black and other." which 
is frequently used to represent the "minority" workforce. One must be careful, therefore, in 
making inter-state comparisons and in using national unemployment data for comparisons with 
state data (and even more so in the case of local data). 

What we are addressing here is one of the most difficult, yet significant issues in program 
evaluation The decision that is made will determine the entry for checklist Item II. "Findings 
(results of data analysis, .warranted inferences, generalizability). Part of the findings were 
reported at the beginning of this section: unemployment rates of 8.9% and 9.8% calculated for 
minority vocational completers in two states. But what answer can be given to the question 
posed at the outset: Does completion of a high school vocational program reduce the risk of 
unemployment for minority youth? What inferences are warranted from the data on vocational 
completers and the comparison group? Is State-A's 8.9% unemployment rate for minority 
completers m early 1979 a good outcome, a bad one. or somewhere in-between? 

Varrous unemployment rates for State-A wer( listed above (a.b.c.d.e) for possible »'se in 
making comparisons between vocational and nonvocational students. The 8 9% rate for mincnty 
completers (UR^^ ) clearly is much higher than the 3.2% rate for the overall State-A work force 
(UR3) 1 he latter, however, reflects the labor force status of prime-age workers (whose jobless 
rate is consistently much lower than the youth unemployment rate) and non-minority w^rkers 
(whose UR. again, is below that for minorities). Comparing UR^ with UR^ is therefore ot very 
meaningful 

The 8 9% January-February 1979 unemployment rate for minority completers is higher than 
the rate for all minority workers in the state (UR^^ - 6.1%. 1978 annual average) but again the 6.1% 
rate measures the labor force status of prime-age minority workers as v^ell as youth, so this 
comparison too is of limited value 



■ Percent not shovvri by BLS where base is less than 75.000. calculated from BLS data with a 
base of 63.000 Data on all 1978 high school graduates (not enrolled m college plus those 
enrolled m college) showed unemployment rates of 10 3% for whites. 40 1% for blacks, and 16 2% 

Hispanics Source BLS. 1979. Table 3 The Special Report was reprinted, with suppiemontary 

.PS from the October 1979 issue of [hv Monthly LahOi Review 
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The 8 9% rate is also lower than the jobless rate for all youtn (UR^ = It. 8%, 1977 annual 
average) and even more so for minority youth (UR^^ "16 4%," 1976 annual average) The time 
periods to which the rates refer, however, differ significantly. Indeed State-A's overall 
unemployment rate m t976 (UR *■ 6 4%) was exactly double the February 1979 figure. Moreover. 
UR^ and UR^| measure the labor force status of high school dropouts and teenagers still enrolled 
in high school (groups tending to have the highest of all jobless rates) as well as high school 
graduates Comparing UR^ with UR^ and UR^ is therefore not really appropriate. 

One of the possibilities that remains is to compare data in State-A with national data. Two 
major factors argue against making direct comparisons. First, State-A's unemployment rate in 
early 1979 was only half the national figure, and in fact was at or below the frictional 
unemployment level that economists use to define full unernpluyrnent/'* At this low level, the UR 
toses much of its capacity to measure involuntary unemployment. Secondly, there are no 
national figures on "minority" unemployment. To use national data would require calculation of a 
weighted average on the basis of the mix of black, Hispanic, and other minority groups, or 
simply the use of the unemployment rate for Hispanics as a proxy for the entire minority 
population Considering that the category Hispanic includes workers of Puerto Rican, Cuban. 
Mexican. Spanish, and other origin this procedure does not appear appropriate for a state that 
may have few if any workers from some of the groups indicated. 

Another possibility is to compare the 8.9% unemployment rate for minority completers with 
the 5 0^*0 rate for all completers ( UR^) This comparison "controls" for time (common survey 
period), age. and educational attainment (as well as the variables of sex and labor market area, 
to some extent) What is not controlled is race/ethnicity. 

The ratio of the unemployment rate for minority completers to the unemployment rate for all 
completers to the unemployment rate for all completers (8.9°/o/5.0%— which we term the 
unemployment disparity index (UDI)*~-is 1.8. This ii^ute indicates that completing a high school 
vocational program does not bring the risk of unemployment for minority youth down to parity 
With non-minority youth (Such a situation would yield a UDI of 1.0.) Few would predict that 
result To the extent that racial^ethnic discrimination and related differences exist in labor 



Frictional unefiipioy rnent is causefi by worKors being "between jobs " or. as in the case of 
young people or displaced homemakers. looking for their first jobs. Workers may be fnctionallv 
unerriployed because of bad weather, model changeovers. or other temporary interruptions in 
production. Fo/the economy as a whole the level o^ frictional unemployment is thought to be m 
the range from throe to five percent Because of initial job-search problems and greater mobility. 
y()ut^l probably have a hiqhpr level of fnctional 'jnemploymont than the pnmo-aqe labor force 
!>pe McConnoII (1978) 

' The ernpioyment disparity inde^ (UDI) is defined as the ratio of the unemployment rate (UR) of 
ono labor force subset (M) to the rate for the entire labor force { 1 ) or. alternatively, to the 
(:(;mp|pment (J) of the first subset (1 e . J - M) Thus. UDI UR^orUDI^ UR^ 

liR. UR 




ma'-Kets. one would hardly expect that a few hundred hours of high school training could 
completely offset mr^rket inequities But one mi(^ht ponder whether completion of an approved 
hiqh school vocational program would reduce the disparity in the risk of unemployment. 
Suppose the unemployment rate for all minority youth in the state is 20% (i.e.. one minority youth 
in every five can expect to be jobless) while the figure for all youth is 10% (i.e.. UDI = 20%/IO% = 
2 0). If the comparable rales for minority vocational completers— most of whom are also high 
school graduates— and all vocational completers were, say 12% and 8% (i.e.. UDI = l2%/8% - l.sf 
then It could be inferred that vocational education may have reduced the relative risk of ' 
unemployment for minority youth. 

In the absence of directly comparable unemployment data (i.e.. to show the influence of 
completing a vocational program on the risk of unemployment as opposed to the effects of age 
and high school graduation), UDIs were calculated and applied to the i979 follow-up data. 
Results showed lower UDIs for all workers (minority UR of 6.1% in 1978 J total UR of 3.7% in 1978; 
UDI 1.6) and for all youth (minority UR of 16.4% in 1977 ? total UR of 11.8% in 1977; UDI = 1.4) 
than for minority vocational completers compared with all vocational completers. Thus, while the 
combination of characteristics A (completion of a vocational program) + B (graduation from high 
school) ^ C (becoming 19 years of age) taken altogether may or may not reduce the absolute risk 
of unemployment for minority youth in Stato-A. there is no evidence that factor A alone lowers 
the relative risk of unemployment for this group in the short run. In the concluding section of 
this chapter, additional observations will be made about the unemployment dispahty index and 
possibilities for further development and use of the disparity index concept. 

Turning to State-B. less comparison data was found there than in State-A. As reported in 
Table 6 above, the unemployment rate calculated for minority (i.e., black) completers was 9.8%. 
The UR for all completers was 4.7%. and for white completers it was 3.2%.. The latter yielded an 
unemployment disparity index of 3.1. which compares with a UDI of 2.6 fo} all black workers in 
the state versus all white workers (1978 annual average). Since there were no recent state data 
for minority youth, it was not possible to calculate directly a UDi for the youth subset of 
State-B's labor force 

Use of national data, however, appears to be much more appropriate for making 
comparisons in State-B than State-A, chiefly because of the larger size and greater homogeneity 
of State-B's minority population. The BLS estimates for black 1978 high school graduates not 
enrolled m college (national sample) indicated an unemployment rate of 39.7%. In 1977 und 1978 
State-B's estimated unemployment rates for teenage black youths and for black workers of all 
ages (the only two age categories of black workers available)— including high school dropouts, 
enrolled students, and graduates— were about the same as national rates. This suggests that 
national data can be used as a proxy for State-B data. State-B's UR of 9.8% for vocational 
completers from the same high school cohort was less \\^an one-fourth the national rate, 
suggesting d reduction of more than 75% in the absolute risk of unemployment for 3tate-B's 
minority youth It is not clear how much of this risk-reduction can be attributed to completing a 
vorational program and how much might be associated with other factors that make up the 
State-Ei education and employment picture 

Regarding the relative risk of unemplo/ment. the BLS national data indicated a UDI of 3.8 for 
black Migh school graduates as compared with white graduates Since the UDI for black versus 
whitp vocational completers in State-D was 3 I. the data indicate a reduction of 18% in the relative 
fi.sk (;f unemploymen? for black youth It seems warranted to infer, therefore, that the nsk of 
unemployment for minority (i e . black) youth m State-B is substantially reduced by completing a 
high school vocational program See Table 7 for summary data on both states 
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Table 7 



MEASURES OF UNEMPLOYMENT DISPARITY 
AMONG HIGH SCHOOL VOCATIONAL COMPLETERS 
AND OTHER GROUPS OF WORKERS: SUMMARY TABLE 



State 


Group 


Data Source and 
Time Period 


Unemployment 
Rates 


Unemployment 
Disparity Indexes ^ 


A 


Minority 
Compljters 

All r^r\p>^ rxlotorc 
Mil v^UiTipit^ 1(71 b 


State Vocational 

Follow-up 

as of Early. 1979 


8.9% 
vs 

J. / /o 


UDi^= 1.8 


A 


Minority 
Workers 
vs 

All Workers 


State Employment 
Security Agency, 
1978 Annual 
Averages 


6.1% 

vs 
3.7% 


UDI = 1.6 
1 


A 


Minority 
Youth 
vs 

AH Youth 


SESA, 
1977 

Annual 
Averages 


16.4% 

vs 
1 1 .8% 


UDi^= 1.4 


B 


Black 

Completers 

vs 
White 
Completers 


^tatp Vnrationfil 

Follow-up Survey 
as of Early 1979 


9.8% 

vs 
3.2% 


\J U * O.I 

2 


B 


Black 

Graduates of 
1978 High School 
Class 
vs 

White GracJuatus 


U.S.- Bureau of 
Labor Statistics 
as of October. 
1978 


39.7% 
vs 

10.5% 1 


UDI 2= 3.8 



S«e text for a definition of the unumployment cJisparity index aficJ for alternative formulations. 
SOURCE Ta[)U} 5, Tdt)le 6, text, and footnotes. 
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Not to bo overlooked in a discussion of findings aro the data on high school vocational 
completers as a whole, i.e . minority plus non-minority. For all completers in State-A, the 
unen^iployment rate in early 1979 was 5.0%. and in State-B the unemployment rate was 47% (see 
Tables 5 and 6 above). These figures approximate frictional levels of unemployment, indicating a 
zero level of involuntary unemployment. Depending on the frame of reference, these findings 
could be regarded as more significant than the findings reported fgr minority students. 
Evaluation findings depend on what questions are asked and what evaluatior) standards are 
applied as well as on the data Again, however, since the outcome selected for analysis here was 
defined in terms of minority students, no further discussion will be presented on the broaden 
category of all completers. 

The impact of these findings (checklist Item 12) and the manner and extent to which they are 
disseminated (Item 13) will largely be determined by officials of the two states. As previously 
stated, the National Center's piimary concern in this R&D study is the identification of outcome 
criteria and development of operational procedures ratner than carrying out and reporting 
in-depth empirical evaluation studies per se. 

With respect to the particular outcome used in the pilot tests, several implications for further 
research, development, and evaluation (HD&E) activities can be noted (checklist Item 14). First, 
some of the findings, especially in State-B, appear to differ from earlier studies. Methods and 
results should be carefully compared. Second, steps have already been taken to replicate the 
study of this outcome in a third state, with certain modifications reflecting local interests and 
concerns. 

In State-A the question was raised as to whether the evaluation standard should require a 
/owe/- unemployment rate for minority vocatiunal students than for minority nonvocational (i.e., 
college preparatory and general) students of the same age and educational attainment. If 
vocational students tend to have socioeconomic and demographic characteristics that place 
them at a disadvantage with nonvocational students in the competition for jobs (controlling for 
race/ethnicity). then it should suffice as a demonstration of vocational education's effectiveness 
to overcome the initial disadvantage (partially or completely) without requiring that vocatiorxal 
students out-perform nonvocational students. Data on the characterisitics of vocational versus 
nonvocational students, however, did not exist. Consequently as a byproduct cf the pilot testir^g. 
an internal study was undertaken to compare verbal and quantitative abilities of a sample of 
vocational and nonvocational students. Interest was also expressed in a follow-up study of the 
completers who were reported to be unemployed at the time of the survey in order lo learn why 
they were jobles;:> because they were not offered employment; because they were holding out for 
a )0b relatf>d to their occupational training; or for other reasons. 

One final implication for further HD&t* activities. If resources and time vv'ere available, it 
might be possible to develop better comparison-group data than presently exists within the two 
states Special follow-up surveys of nonvocational students in the same high school cohort as 
the vocational completers could be conducted, thereby controlling for labor market conditions as 
well as age and educational attainment 



C. LImlteMons of the Pilot Tests 

D>.?spite limitations in the pilot testing, some important lessons have been learned about data 
availability, policy and program differences among SliAs and Ltf.As. complexities and 
unanswerables related to evaluation standards, and the practical usefulness of the overall 
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procedures employed. A yroat deal more could bo learned if SEA, LEA. and independent 
evaluators were to conduct outcome studies using procedures similar to those described in this 
report subsequently sharing their findings and procedures with other interested evaluators. 
Program evaluation, however, is a painstaking, time-consuming, and costly activity when 
carefully done. The "quick-and-dirty" approach may save time and money but is unlikely to yield 
credible results that can serve as a basis for informed decision making or add to the general 
body of documented knowledcje concerning the outcomes of vocational education. 

In the preceding section, reference was mnde to the concept of the unemployment disparity 
mdex whi;:h was developed as one possible aps^roach to providing quantitative evaluation 
standards. The use of ratios, particularly if they are reasonably constant over time, can help the 
evaluator make meaningful comparisons even when data are not available for a particular time 
period or population. The basic concept underlying the unemployment disparity index has many 
potential applications. As is true with any st&itstical tool, it is also subject to limitations and 
vulnerable to misuse. Furthor developmental work should be done with disparity indexes before 
the technique can be unreservedly recommended for general use. 

Another important limitation of the pilot studies concerned the definition of the strategic 
independent variaole (checklist Item 4 in Tables 3 and 4). The project staff suspects that 
significant differentials would be found to exist in unemployment rates according to the 
particular occupational programs (e.g.. Distributive-^Education, Trade and Industrial. Business 
and Office) that various minority students completed. In this study, all programs were lumped 
together under the rubric of vocational education. 

Finally, it may bear repeating that the entire study was limited lO the topic of outcomes. No 
attempt was made to evaluate vocational education on the basis of other types of criteria (see 
Chapter I). Nor were cost factors or monetary benefits addressed iri the study. The focus was on 
program effects, not program efficiency. 
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V. SUMMARY OF FINDINGS AND RECOMMENDATIONS 



This chapter presents a summary of findings from the National Center's two-year study of 
outcomes and concludes with recommendations for utilizing the products of this study in making 
policy decisions and carrying out further research, development, and evaluation (RD&E) 
activities. 



A. Some Lessons Learned from the Study 

Phase One of the outcome study, initiated in 1978, was essentially exploratory. The project 
staff took stock of where vocational education was and where it had been with respect to 
outcomes evaluation. An examination of evaluation literature disclosed conceptual ambiguities, 
data gaps, and methodological shortcomings. Published and unpublished reports questioned 
whether vocational programs had actually improved occupational skills, employment success, or 
wage rates. One writer complained that '^analysts have not even agreed on . . . (vocational 
education) outcomes to be tested" (Reubens, 1974, p 24). The staff concluded that research and 
evaluation had not yet produced a substantial body of documented knowledge regarding the 
outcomes of vocational education. 

Some additional lessons learned by the project staff during early stages of the inquiry and 
confirmed in the second year were the following; 

• Members of the vocational education community, including researchers and evaluators, 
tend to use the terms outcomes, outcome measures, program goals, and program 
benefits interchangeably. This practice interferes with clarity of communications and 
clarity of inquiry. 

• In many reports and discussions it is not made clear what is being evaluated— an 
outcome, a group of students, a vocational education program, or something else. In 
particular, there seems to be persistent confusion about the role that outcomes play as 
evaluation criteria. 

® Views held by many members of the vocational education community, policy analysts, 
and others disclose a lack of appreciation of the broad range, diversity, and complexity of 
possible outcomes "Placement" in a job that is related to a former student's area of 
occupational training is sometimes viewed as the sole worthwhile consequence of 
vocational education, even at the high school level (although this position in by no means 
universally accepted by members of the vocational education and RD&E communities). 
Evaluation standards typically are not cleanly identified, and in fact are often completely 
ignored in what purport to be evaluation studies Yet if there is no standard to which 
program outcomes can be compared, there is no basis for formulat ng judgments 
concerning the merit of the program, i.e . there is no real evaluation. 
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• Thor<) seemed to bo no clear conceptual base for program evaluation, no consensus on 
procedures, and little agreement on research techniques and statistical methods? to bo 
employed. The relative importance of outcomes compared with other types of evaluation 
criteria has not been well addressed. 

• Many evaluation studies have strong overtones of advocacy. They seem to be motivated 
less by a desire to evaluate programs on the basis of empirical evidence and cogent 
evaluation standards than by a zealous deiermination to either document program 
benefits or to "prove" that vocational education does-inot generate benefits. 

These perceptions influenced the staff in setting and adjusting priorities for its work 
throughout the two years of the inquiry. 

Responding to some challenges implied by the state of affairs described above, the National 
Center staff has developed 

(1) A compe.idium^f 252 vocational education outcome questions (Year i) 

(2) A selected list of fifteen key outcomes for possible use, among other cnteria. in 
evaluating vocational programs (Year 2) 

(3) A conceptual framework for outcomes evaluation and clarification of outcome 
evaluation concepts (Years I and 2) 

(■1) A general set of pilot-tested operational procedures for evaluating vocational 
programs on the basis of outcomes; findings and limitations of the pilot tests 
are presented in Chapter IV (Year 2) 

(5) Procedural notes for operationalizing each of the fifteen selected outcomes according 
to a consistent format (Year 2). 



B. Recommendations 

The following recommendations for applying the lessons and products of this study are 
addressed primarily to vocational oducation evaluators, supervisors of evaluation, state and local 
vocational directors, and sponsors of evaluation studies. The recommendations may also be of 
interest to policy analysts, university instructors and researchers, and to people who advise and 
make decisions on vocational education policy. 

(I) In planning evaluation studies, care should be taken to determine clearly what is to be 
evaluated and what criteria, data, and evaluation standards are to be used. There are 
numerous potential candidates to serve as the object of an evaluation (including 
broadly defined or more narrowly limited vocational programs), many and diverse 
criteria, and a variety of data types and evaluation standards. Establishing agreement 
on criteria and standards may not be an easy task The fourteen-point checklist of 
essential elements of information is no magic formula; but its use is recommended for 
designing and carrying out evaluation studies where clarity, completeness, and 
consistency are desired 
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(2) It should be openly acknowledged that no national consensus exists on a limited set 
of outcomes to serve as evaluation criteria. There are disagreements concerning the 
appropriateness, importance, and feasibility of various outcomes. Outcome criteria 
that are appropriate for State X in Year Y may not be at all suitable for City R in Year 
Z. Sponsors of evaluation studies should provide for the selection of criteria and a 
corresponding rationale to justify the selection in the particular context of the study. 

(3) It should also be recognized that no single outcome criterion is likely to be 
satisfactory by itself for evaluating a vocational program. Educational programs 
generate multidimensional outcomes. What is viewed one year as a principal product 
of vocational education may in two or three years be relegated to secondary 
importance, with last year's "byproduct" emerging as a new favorite accountability 
criterion. Evaluation sponsors are advised to avoid putting all of vocational 
education's evaluation eggs in one small, rigid basket. 

(4} The concept of the unemployment disparity index (UDI) introduced in Chapter IV. 
and similar techniques for operationalizing evaluation standards, should be further 
developed and carefully critiqued. 



(5) Implementation of the RD&E agenda proposed in an earlier report (Darcy. 1979a) 
should be vigorously pursued by vocational education organizations including the 
National Center. U.S. Department of Education. National Advisory Council on 
Vocational Education, and state and local education agencies. Sustained efforts are 
needed over the next three to five years to 



• resolve definitional problems 

• clarify the mission and objectives of secondary, postsecondary. and adult 
vocational education 

• define data needs and collect key data 

• expand awareness of the diversity and range of possible outcome criteria 

• provide specific evaluation standards 

• accumulate a documented body of empirical knowledge concerning outcomes, and 

• assure that evaluation findings are (a) clearly communicated to relevant audiences 
(mcluding policy makers, practitioners, and other evaluators) and (b) applied in 
the interest of improved educational effectiveness and efficiency. 



(6) Finally, efforts to expand our knowledge of the actual outcomes of vocational 
education should be directed not only to labor market conscsuences but also to 
outcomes of a nonmarket nature. 



Effective dissemination and application of procedures described in this report could have a 
positive impact on vocational education evaluation in the 1980s. It has also been suggested that 
the ideas and information presented here may have a broader influence— by changing the way 
people think about other facets of vocational education. Increased awareness of outcome 
evaluation issues could stimulate a systematic reexamination of educational goals, needs and 
procedures for program design and management, staff development, and policy making at 
community, state, and national levels 
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